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ABSTPACT 

This report contains the recommendations of the 
Advisory Committee on Program and Cost Effectiveness to the 
California State Board of Education. The committee was established by 
the California legislature for the purpose of developing a 
methodology for evaluating the cost effectiveness of various Federal 
and State educational projects. The report defines cost effectiveness 
as a management decision piccess that allows the decisionmaker to 
choose from among feasible alternatives on the basis of least costs 
and maximum program output as measured by student achievement. I'ive 
components were deemed to be esse^itial to the development of a 
methodology for determining the cost effectiveness of educational 
programs: establishment of terminal objectives by grade for each 
program, measurement of pupil achievement in relation to specified 
objectives, determination of student characteristics that affect 
learning, identification of costs, and development of a system for 
the documentation of all program elements. The experience of the 
committee has convinced its members that it is indeed possible to 
develop a methodology for the assessment of the cost effectiveness of 
educational programs. A bibliography is included at the end of the 
report. (Author/DN) 
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PINAL REPORT 
of th<3 

ADVISORY COMMITTEE ON PROGRAM AND COST EFFECTIVENESS 

as established by the 
Education Improvement Act (AB 606) 

of the 

California State Legislature 
of 1969 



March 1972 



August. 1, 197: 



The Honorable Wilson C. Riles 
Superintendent of Public Instruction 
California State Department of Education 
721 Capitol Mall 
Sacramento, California 95814 

Dear Dr. Riles: 

It is my pleasure to submit to you the second and concluding report 
from the Advisory Committee on Program and Cost Effectiveness. This is a final 
report only because of the termination of this committee in accord with the 
provisions of AB2800, not because the assigned task has been completed. 

The conclusions reached were those of informed lay citizens, who 
attempted to tap many sources of professional education, finance and business 
knowledge and opinion in California and to take account of the concerns of all 
Californians for the improved evaluation of education for California's children 
and youth and the resultant improved use of tax monies. 

Our report, therefore, is a mixture of lay and professional judgment. 
The recommendations should be regarded as broad principles for the direction of 
cost effectiveness in public education. Any implementation effort will need the 
continuing attention of the Board, the Department of Education, the Legislature 
and the education communities. 

The Committee of Nine was announced March, 1970. It has devoted long 
hours to its work and as personal situations changed, several members were 
obliged to withdraw. The Committee wishes to thank the State Department of 
Education for its increasing assistance and cooperation in this endeavor. 
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I'ho Honorable Wilson C. Riles 



Page Two 



By submitting this Final Report and with the passage of AB2800, the 
Advisory Committee on Program and Cost Effectiveness has completed the charge 
it received from the State Legislature and the State Board of Education, 

With our sincere thanks for the opportunity to serve the people and 
children of California, we remain, 




Members: John A. Geddes, Vice Chairman 
Allen S, Ginsburgh 
Donald E. Kitch 
Lewis T. Kohler 
Agnes S. Robinson 
Clarence R. Newby 



cc: President 

State Board of Education 
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STATE OF CALIFORNIA 

DEPARTMENT OF EDUCATION 

STATE EDUCATION BUIIDING, 721 CAPITOL MALL, SACRAMENTO .95814 

February 23, 1972 ^ 



The Honorable Wilson C. Riles 
Superintendent of Public Instruction 
California State Department of Education 
721 Capitol Mall 
Sacramento, California 95814 



Dear Dr. Riles: 



I am pleased to submit to you the final report from the 
Committee on Program and Cost Effectiveness. This is a 
final report only because of the termination of this 
committee in accoid with the provisions of AB 2800, not 
because the assigned task has been completed. 

This report is a brief summary of the work done by the 
committee since its inception in March 1970, and con- 
tains our conclusions and recommendations for work yet 
to be done. A more complete documentation of the work 
of the committee is being prepared for those who wish to 
examine in more detail the rationale and activities that 
have led us to the conclusions expressed in this report. 

The committee has been composed of three members repre- 
senting business management, Mr. Morris Currey (resigned 
May 1971), Mr. Allen Ginsburgh and Mr. Leo Newsome 
(resigned November 1970) ; four members representing economics 
and finance, Mrs. Dorothy Dehr (resigned June 1970), 
Dr. Lewis Kohler, Mr. Clarence Newby, CP. A. and 
Mr. Alvin Rosenthal, C.P.A. ; and three members from the 
learning sciences. Dr. John Geddes , Mr. Donald Kitch and 
Dr. Agnes Robinson. As chairman I am indebted to them for 
their time and conscientious labor. 



The committee wishes to thank the State Department of 
Education for its assistance and cooperation in this 
endeavor. 



^^^^Res^ec^ully^^s^^^ , 

' '^t^/iB ^I^Ros e n tl'ia^^^^fea irma 
Advisory Committee on Program 
and Cost Effectiveness 
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REPORT TO THE STATE BOARD FROM THE ADVISORY COMMITTEE 
ON PROGRAM AND COST EFFECTIVENESS 



The Legislature established the Committee on Program 



and Cost Effectiveness by the passage of AB 606 (California 
Legislature, 1969) and defined its charge in AB 1923 



DEFINITION OF COST EFFECTIVENESS 

Cost effectiveness is a term that has been used to describe 
several different, and sometimes conflicting, concepts. In the 
context of this report, cost effectiveness has been defined as 
a management decision tool which allows the decision maker to 
choose from among feasible alternatives on the basis of least 
costs and maximum program output as measured by student achieve- 
ment. This may take the form of a ratio or index which can be 
used to compare costs for achieving the same objectives by the 
use of different programs. 

POTENTIAL BENEFITS 

The potential benefits to education that could result 
from the completion of the methodology started by this com- 
mittee would include the capability to make decisions based 



(California Legislature, 1970). The charge reads as follows: 



**lhe Advisory Committee on Program and Cost 
Effectiveness shall develop and recommend to 
the State Board of Education a methodology 
for evaluating the cost effectiveness of 
projects financed by Titles I and III of the 
Elementary and Secondary Education Act of 
196^^, Hi llez'Unruh Basic Reading Act of 
1965 ..." 




on: 

1. The relative cost effectiveness of programs 
with the same objectives when a cost effec- 
tiveness index has been developed. 

2. Objective data related to costs of programs 
and program elements. 

3. Objective data related to the effectiveness 

of different programs with the same objectives. 

4. The impact of a program on different student 
populations. 

5. Staffing requirements in terms of teacher 
characteristics, skills or teaching styles 
required. 

6. The evaluation of different instructional 
strategies . 

7. The results of replicating successful pro- 
grams or program elements in other locations. 




COMMITTEE ACTIVITIES 

Committee activities for the first year of operation were 
reported to the Board in May 1971, at which time five compo- 
nents were identified which were deemed to be essential in the 
development of a methodology for determining the cost effec- 
tiveness of educational programs. These five components were: 

1. Establishment of terminal objectives by 
grade for each program to be assessed. 

2. Measurement of pupil achievement in rela- 
tion to the specified objectives of the 
program. 

3. Determination by appropriate research methods 
of the student characteristics which affect 
the learning rates of pupils. 

4. Identification of costs, both direct and 
indirect, or programs and program elements. 

5. Development of a system for the documenta- 
tion of all program elements such as course 
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content, instructional strategies, staffing, 
inservice education, etc. 

A more complete jlescription of these components was given in 

this committee s report to the State Board dated May 25, 1971. 

The report concluded that it was the opinion of the 
committee that without the development of these constituent 
parts, a model , as such, could not be developed. 

During the second year the committee attempted to develop 
and test these components. In July 1971, in order to provide 
a working laboratory in which to try out some of the concepts 
contained in these components, the committee negotiated with 
the Riverside Unified School District to operate a reading 
and mathematics program in grade 7 at the Central Junior High 
School. This was a program Riverside operated at the school 
during the school year 1969-70 under the provisions of AB 938, 
the Demonstration Program in Reading and Mathematics, and the 
same source was usied to fund thia program for the second time. 
It was the intent of the committee that in the operation of 
the project. Riverside would collect data appropriate to each 
of the five components that had been identified by the com- 
mittee and document carefully all aspects of the program and 
its implementation. This would enable the committee to test 
the feasibility of gathering these types of data at the 
school and classroom levels. It would also provide addi- 
tional data for the study of student characteristics affecting 
the learning rate of different groups of students as well as 
data indicating the potential importance of measurable 
teacher characteristics. Also, since Riverside previously 



operated this same program, the degree o! replication could 

be documented, as there would be a basis for comparison with 

the original version. The program involves the total grade 7 

population of approximately 370 pupils at Central Junior High 

School in the subject areas of reading and mathematics, and 

includes the Chairmen of the English and Mathematics 

Departments, eleven teachers, teacher aides, tutors and staff ^ 

time from the busine^cs office and Research and Evaluation ^ 

unit. A budget of $84,469 was approved to cover actual costs ^ 

over and above regular program expenditures. This proiect 

is currently in operation and is scheduled to run through ^ 

the 1971-72 school year. 

CONCLUSION 

The experience of the committee over the past two years 
has co:r^inced its members that it is indeed possible to develop 
a methodology for the . assessment of the cost effectiveness of 
educational programs. 

The methodology as presently conceived by the committee 
would require definable program elements including objectives, 
measurement methods and a full description of such areas as 
subject content and teaching strategies; any measurable or 
definable teacher characteristics or teaching styles; any 
measurable student characteristics that affect the learning 
rate of different atudenf?; program costs down to the units 
within the classroom. 

Before such a methodology may be completed, a number of 
tasks remain to be accomplished. Work could proceed 
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independently on the first four recommendations listed 
below, but before the last two could be completed > it would 
require the stabilizing of program content and implementa- 
tion procedures to ensure comparability of programs. 

CONCLUSIONS AND RECOMMENDATIONS 
Conclusion No. 1 

Valuable data are being collected in the Riverside 
project and will become available following the close of 
the school year 1971-72. 

Recommendation No. 1 

Upon receipt of the final report from the Riverside pro- 
ject, the findings and conclusions should be carefully studied 
and incorporated into the development of the proposed model. 

Conclusion No, 2 

Programs must have the same objectives if they are to be 
compajTed for relative effectiveness. 

Recommendation No. 2 

If programs are to be compared, they must have the same 
objectives and be measured by the same instruments. In order 
to broaden the base for program comparison, it is recommended 
that when the Board adopts a curriculum framework, objectives 
for that framework and appropriate instruments to measure 
those objectives should be identified concurrently. 
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Conclusion No, 3 

when a successful program has been identified , it is 
possible to replicate that program in another location. 

Recommendation No. 3 

It is recommended that research be undertaken to identify 
the elements of a program of instruction that need to be com- 
pared in order to determine if two programs are the same or 
different, and that a format be developed for the collection 
of data appropriate for the elements identified. It is fur- 
ther recommended that procedures for data collection and 
program replication be field tested until it has been experi- 
mentally established that the proper data are being transmitted 
and the proper procedures are being used in the process of 
replication so that a successful prog^rarci Deing operated in one 
district may be duplicated in another district without signi- 
ficant distortion of the original program. 

Conclusion No, 4 

It is possible to develop a cost accounting system that 
will allow an analysis of costs down to the level of units 
within a classroom. 

Recommendation No. 4 

The accounting system developed for the Riverside pro- 
ject should be examined, revised as needed, and further field 
tested in order to ensure that the system is generally appli- 
cable to other districts and that it is collecting the kinds 



of data required for replicating programs and comparing rela- 
tive costs. 



C onclusion No. 5 

With a given program, it is possible to calculate and 
subsequently predict the varying impact or effectiveness of 
that program on student populations of varying backgrounds 
or characteristics , 

Recommendation No. 5 

When the replication process has been tested sufficiently 
to ensure duplication of a program without significant distor- 
tion, a successful program should be applied to a sample of 
student populations. This sample must be representative of 
the total state in regard to each of the variables identified 
as characteristics which affect the learning rates of pupils. 
The responses of students with particular characteristics to 
a program of known dimensions could be treated statistically 
to forecast the effects of this particular program on other 
students with these same characteristics. 

Conclusion No. 6 

It is possible to predict the relative impact a known 
program will have when Implemented by teachers with varying 
backgrounds or characteristics. 

Recommendation No. 6 

It is recommended that research be undertaken to identify 



teacher characteristics or teaching styles that enhance or 
inhibit the ability to successfully implement particular 
educational programs, and that these results be either de- 
fined as an important element in the program description 
or incorporated in the model for the assessment of cost 
effectiveness, 

FUTURE SUPPORT 

It is the final recommendation of the Committee on 
Program and Cost Effectiveness that the State Board of g 
Education exercise leadership in the further development 
of the concepts presented by the committee and that suffi- 
cient manpower and financial resources be provided to ensure 
the completion of this methodology for the assessment of 
program and cost effectiveness within a reasonable length 
of time. The committee has made a recommendation to the 
State Superintendent of Public Instruction that a specific 
work group be formed within the Office of Program Evaluation 
of the State Department of Education for the purpose of con- 
tinuing the work of the committee. 



a. 

s 



Alvin L. Rosenthal, Chairman 

John A. Geddes, Vice-chairman 

Allen S. Ginsburgh 

Donald E. Kitch 

Lewis T. Kohler 

Clarence Newby 

Agnes S. Robinson 
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LEGISLATION AND LEGISLATIVE INTENT 



AB 606 Of the California State Legislature, 1969, contained a 
chapter known as the Educational Improvement Act of 1969, 

It was the intent of the Legislature that funds provided through 
several specified sources be spent in the most effective way pos- 
sible and have cost effective measures employed in their approval 
and evaluation. Program achievement and cost effectiveness were 
to be assessed annually. Highly effective projects were to be 
extended to further use within the district where it was originated 
as well as in other districts, while less effective projects were to 
be replaced with ones of proven effectiveness or by new promising 
projects. 

The effectiveness of a project was to be measured in terms of the 
objectives of the project which were to be primarily concerned 
with the pupil's improvement in ability to read, to use and under- 
stand the English language and to use and understand the concepts 
of mathematics. 

An important part of the Educational Improvement Act was the crea- 
tion of an Advisory Committee on Program and Cost Effectiveness, 
This Committee was to be appointed by the State Board of Education 
and was to consist of three public members representing the field 
of economics, three public members representing the learning 
sciences and three public members representing the managerial 
sciences. The role of this Committee was to advise the State 
Department of Education and the State Board of Education on proj- 
ects to be approved and the administration of the several speci- 
fied acts. The Committee was also to assist in the evaluation of 
the program achievement of projects, the relative cost effective- 
ness of projects and in the selection of projects which should 
have expanded use or which should be modified or replaced. 

Later legislation, AB 1923 of the California State Legislature, 
1970, revised the charge to the Committee and required that the 
Advisory Committee on Program and Cost Effectiveness develop and 
recommend a methodology for evaluating cost effectiveness and 
that the State Board of Education should use such methodology in 
determining which projects should be expanded, modified or 
replaced. 

The advisory nature of the Committee was well stated by the 
Chairman, Mr, Alvin L, Rosenthal, in a memorandum to the Committee 
members which stated in part: 

. , This memorandum is to offer for your consideration my 
thoughts and the thoughts of others who have set forth in print 
a concept and function of an Advisory Committee, 

Considering the word advisory, there is a distinct implication 
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that the individual or individuals so charged will consider and 
weigh a given amount of data or set of circumstances or both and 
from this draw conclusions in the form of opinions or advice. It 
follows naturally that these opinions or this advice will be 
relayed to a pre-designated individual or group who will in turn 
weigh the quality of the advice as veil as the knowledge, reputa- 
tion and integrity of the advisors. The individual or group having 
thus been advised needs only to act upjn that advice. 

Advisory Committees are set up to respond to a great variety of 
technical and scientific problems. The members of the group are 
invariably selected because they possess specialized knowledge, 
which when brought to bear on the problem should bring forth the 
quality of advice that is being sought. 

It would appear to me that the members of most Advisory Com- 
mittees already possess the knowledge required to render the 
advice being sought. In the situation where the task is in the 
nature of a research project, however, the members of the Committee 
might in all likelihood spend considerable time learning the 
present "State of the Art." They might also spend some time in 
maintaining the requisite knowledge to render competent advice. It 
would appear to me, however, that beyond learning the "State of the 
Art" and maintaining the requisite knowledge, the members of an 
Advisory Committee need only to mobilize the forces required to 
undertake £md accomplish the research or development task. 

Consider this: If an Advisory Committee were formed to bring 
into existence a very specialized type of building for the Govern- 
ment or for a Corporation, its members would no doubt represent a 
variety of professional disciplin<2s. The group would no doubt 
meet and elect a chairman and set about to discuss the needs of 
this building and when it should be completed. After establishing 
a broad-building concept, the Committee would undoubtedly select 
an Administrator who would oversee the entire project. This 
.Administrator would then definitize the Committee's building 
concepts to a degree that the building needs could be interpreted 
by an architect. At the proper point in time, an architect would 
be selected and the plans would be prepared. The Administrator 
would at the proper time consult various prime contractors or 
possibly recommend to the Committee that the job be accomplished 
by engaging subcontractors. If the subcontractor route was 
adopted, the Administrator and the Committee would then proceed 
with the building plans, considering each aspect of the task as 
it came to fore. Where structural, electrical, air-conditioning, 
decorating and dozens or other specialized matters were involved, 
certain members of the Committee might assume a more dominate 
advisory role. At no point in the entire project would any member 
of the Advisory Committee lay a brick, weld a girder, or paint a 
wall, or even give a direct order to a foreman or supervisor on 
the job. Advisory Committees are not formed for that purpose; 
their purpose is to give advice, counsel and direction to their 
administrative officer and to their sponsoring authority in order 
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that the specified task can be accomplished to the optimum benefit 
of the sponsor." 
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ORGANIZATION OF THE COMMITTEE 



The first few meetings were devoted to an orientation of the iob 
?L?«i^f-''"'^^i^^u'^ ^""^ specific programs listed in the' 

Education°'^ ^^^^ administered by the Department of 

At tha first meeting an outside consultant, at the request of the 
Department of Education, presented an analysis of the work to be 
th2%^?^f ''"h^^^/ ^'^^i organizational char?s shoSing 

Ednnf^io^ various units within the Department of 

proposed b^f ™^™h^°"™i^JK^-. ^^^^^ "^^^^^^9 another plan was 

proposed by a member of the Committee which was subsequently 

of'?hf liL'ol trgroS,"^' "^"^ ^^^^^ 

^ent?f ?L°5^'v^''? ''^^ ^^^^ ^^^^'^^ meeting and subse- 

2n ^ni^i.f! ^ P^^'' ''^^ adopted, sub-committees were appointed 

to complete specific tasks. 

3!j!n^n!rj!^®® met monthly with additional special meetings called 
when needed and with sub-committees meeting between the regular 

^v^ntn«^; 7°'' JU® y®'^'' executive committee met In the 

evening before the regular meeting to assist in planning and 
taking care of minor matters, with the start of the second year 
the executive committee was abolished and the total group, feeling 
f^?iJJ!!^ 1°'' "»ore time, met regularly with an evening meeting 
followed by an all day meeting the next day. 

t^^^^?Lm?^fi''^^^!u'' ^^^^^ ^^^^ ^ problem of communication between 
the Committee and the Department of Education caused by the com- 

Snf!5i«^o°^ u !Jith different perceptions of the desired 

outcomes, a change in administration leadership and lack of 
agreement as to some procedural matters. These were pretty well 
solved by the start of the second year and work proceeded more 

In^/tiL ^-K^?-.-^^^ ^^^^ ^ ^^^"^^ initial definition of roles 
and responsibilities at the beginning could have made for a 
smoother operation daring the first year. 

n^' ^V^- the assistance of Mr. Donald Kitch and 

Mccormick, developed a set of guidelines called 
o^^i^L Procedures of the Advisory Committee on 

Program and Cost Effectiveness." This document contained general 
policy guidelines for the Committee as well as specific rules and 
regulations covering the operation of advisory committees, the 

? executive secretary and organizational charts 
showing the relationship between the Committee and the State 
Department of Education and the California State Legislature. A 
copy of this document is in the permanent files of the Committee. 

follow" sub-committees that were formed are listed as 
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ADVISORY COMMITTEE ON PROGRAM AND COST EFFECTIVENESS 



OFFICERS ELECTED 
March 1970 - April 1971 
Mr. Alvin L. Rosenthal, Chairman 
Mr. Allen S. Ginsburgh, First Vice Chairman 
Dr. Agnes S. Robinson, Second Vice Chairmem 

EXECUTIVE COMMITTEE * 
March 1970 - April 1971 
Mr. Alvin L. Rosenthal 
Mr. Allen S. Ginsburgh 
Dr. Agnes S. Robinson 



OFFICERS ELECTED 
April 1971 - March 1972 
Mr. Alvin L. Rosenthal, Chairmen 
Dr. John A. Geddes, Vice Chairman 



*Discontinued after April 1971 



ADVISORY COMMITTEE ON PROGRAM AND COST EFFECTIVENESS 



SUB-COMMITTEES FORMED 
Chairman Alvin L. Rosenthal - Ex officio member of each sub-committee 



Objectives : Dr. Agnes S. Robinson, Chairman 

Dr. John A. Geddes 

Mr. Morris E. Currey 

Dr. Lewis T. Kohler 



Measurement: Dr. John A. Geddes, Chairman 

Mr. Donald E. Kitch 

Dr. Lewis T. Kohler 

Dr. Agnes S. Robinson 



Scaling : Mr. Donald E. Kitch, Chairman 

Dr. John A. Geddes 
Dr. Agnes S. Robinson 



Costs : Mr. Clarence R. Newby, Chairman 

Mr. Allen S. Ginsburgh 
Dr. Lewis T. Kohler 



Planning : Dx. Lewis T. Kohler, Chairman 

Mr. Donald E. Kitch 

Mr. Clarence R. Newby 

Mr. Allen S. Ginsburgh 




WORK PLAN 



An initial work plan developed by Dr. C. W. Stone 
at the request of the Department of Education was 
presented at the first meeting of the Committee 
in March 1970. (A complete copy of this report 
is in the Committee files „) At the meeting in 
May 1970, Mr. Allen Ginsburgh presented another 
work plan and associated time line which were 
subsequently adopted by the Committee. The work 
plan as presented to the Committee is reproduced 
here in full. 



11 May 1970 

To: Members of the Advisory Coinmittee on Program 

and Cost Effectiveness, California State Board 
of Education 

Prom; Mr. A. S. Ginsburgh, Committee Vice Chairman 

Subject: Tentative Committee Program Plan 



I. INTRODUCTION 

A program plan is presented herein for the committee to 
consider in the accomplishment of the task assigned to it by the 
California State Assembly Bill No. 606, Chapter 6.8, "Educational 
Improvement Act of 1969." 

The above noted act states as follows: 

"6499.201 — It is the intent of the Legislature 
that the funds provided by this Chapter and the funds provided 
through Title I and Title III of the Elementary and Secondary 
Education Act X965, the Miller-Unruh Basic Reading Act of 
1965 . . . and Chapter 106 of the Statuses of 1966, first 
extraordinary session, be expended in the most effective way 
possible, and that cost effectiveness measures be employed in the 
approval and evaluation of all projects. It is the further in- 
tent of the Legislature that all projects be evaluated annually 
as to the degree of progreun achievement and the cost ef fectivieness 
produced; that highly effective projects shall be expended to 
further use in the district where operated and in other districts; 
and that less effective projects be replaced with ones of proven 
effectiveness, or by new projects which hold promise of high 
effectiveness. 

It is the intent of the Legislature that the effectiveness 
of a project be measured in terms of the objectives of the 
project, and that each district should be primarily concerned 
with the pupils' improver.-.ent in ability to read, to use and 
understand the English language and to use and understand the 
concepts of mathematics. *■ 

The Legislature intends that each project be evaluated 
annually by the Department of Education to determine and identify 
its relative effectiveness; that each evaluation shall be 
assisted by an Advisory Committee competent to assess the effec- 
tiveness of the results of the project and to make recommenda- 
tions to the Department of Education and to the State Board of 
Education on projects to be expanded in use and those that 
should be modified or replaced to produce greater effectiveness. 
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"6499.204 — The Advisory Committee on Program and Cost 
Effectiveness shall (1) advise the Department of Education and 
the State Board of Education on projects to be approved, and 
the administration of Titles I and III of the Elementary and 
Secondary Education Act of 1965, the Miller-Unruh Basic Reading 
Act of 1965 (Chapter 5.8) commencing with Section 5770 of 
Division 6 and Chapter 106 of the Statutes of 1966, first 
extraordinary session, (2) assist in the evaluation of the 
program achievement of projects, (3) assist in the determination 
of the relative cost effectiveness of projects and (4) advise 
on the projects which should have expanded use and those which 
should be modified or replaced to produce a higher degree of 
program achievement and cost effectiveness . " 

The supportive subtasks considered necessary to the accom- 
plishment of the above charge are described and scheduled herein. 
A sub-committee structure is recommended and supporting staff is 
defined. The program is scheduled so as to provide firm recom- 
mendations to the Department of Education in August of 1971 and 
to have these implemented and functioning in February 1972. 
The program schedule is shown in Figure 1. 

The Committee has already recognized the need for its own 
education prior to formulating the details of its approach and 
conducting its investigations and evaluations. This education 
should be in four areas; 

1. To thoroughly understand the task intended by the 
State Legislature. 

2. To understand the legislative programs in compensatory 
education for which program and cost effectiveness measurement 
methods will be evolved. 

3. To understand how educational funds are budgeted and 
managed at each level of administration and thoroughly understand 
the sources of these funds. 

4. To thoroughly understand the current state-of-the- 
art of educational program formulation, methods of accomplishment, 
and administration, as well v\s methods of measurement and 
evaluation. 

Following this educational phase, the committee must be 
prepared to establish objectives in usable forms, choose methods 
of achieving these objectives, establish scales for the measure- 
ment of accomplishment toward the objectives along with techniques 
for introducing unquantitative factors. These several elements 
must then be combined into a recommended model or method suitable 
for quantifying, measuring and evaluating program and cost effec- 
tiveness. The committee should then verify the usefulness of its 
model and establish a method for implementing the utilization of 
its model. 
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II* OBJECTIVES IN COST EFFECTIVENESS 



Cost effoctivity is a concept devised in the early days 
of Mr. McNaraara's tenure as Secretary of Defense. It recognized 
that there were more reasonable demands on the dollars then 
available to the Department of Defense than could be met and sought 
to establish criteria for projecting the greatest return for the 
money spent, in this instance, the criteria utilized involved 
tne objective of the weapons systems and concerned such statis- 
'^S^i-? ®s*^^^^shed elements of criteria as functional capability 
and kill probability. One of the most effective criteria that 
can be imagined in this regard would be to consider the dollar 
cost to our Defense Agency to achieve effective equivalent dollar 
loss to our adversaries. Another example of cost effectivity is 
the use of cost per pound of payload in orbit which is used as a 
guide for space booster considerations. A more common ii.easure 
of cost effectiveness is the cents per passenger mile or per ton 
mixe used in comparing transportation means. Note that in each 
case, an objective is established in measurable terms such as 

dollar damage to an enemy," "moving people," etc. It is 
apparent that if we are to think in terms of cost effectiveness 
in this undertaking we must clearly establish our objectives. 
Further, the objectives chosen should be such as to lend them- 
selves to as much measurement in quantitative terms as is possible. 

III. PROGRAM PLAN 

The Program Plan (Figure 1) has been prepared for considera- 
tion of the committee which envisions study and formulation in the 
several areas discussed above, tinally culminating in the construc- 
tion of a model (or the modification and acceptance of an available 
model) and the implementation and utilization of this model. The 
plan envisions establishing sub-committees in the several areas. 
These would draw on the Department of Education specialists and 
specialists and consultants from the academic community. Each 
sub-committee will be charged with accumulating and synopsizing 
information in the area assigned for presentation to the full 
committee alovig with recommendations for action and/or acceptance. 
The sub-committee will also sponsor and plan presentations by 
consultants and other experts to the full committee as they might 
deem desirable. 

The Program Plan is based on the assumption that only 
three working sub-committees can be supported at one time. Bach 
IS expected to meet separately and report to the full committee 
as indicated in the Plan. They will take direction from the fuU 
committee during the period of study. Thereafter, the full 
committee will establish the position on that subelement and 
proceed to the next. 

The program is divided into the following phases and tasks* 
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Phase I - Program Review 



Task A 
Task B 
Task C 
Phase 



Task A 
Task B 
Task C 
Task D 
Task E 
Task F 
Phase 



Task A 
Task B 
Task C 
Task D 
Phase 



Task A 
Task B 
Task C 
Task D 
Task E 
Phase 5 



Title I 

Miller-Unruh Act 
Title III 

- Component Studies 



Program Study and Adoption 

Fiscal Studies 

Objectives 

Methods 

Scaling 

Intangible Equivalent 
- Modeling 



Experimental Modeling 
Data Comparison 
Procedural Definition 
Procedural Evaluation 
- Implementation 



Implementation Studies 

Formulation of Recommendations to the State Board 
State Board Study and Acceptance 
Implementation Supervision 

Development of Management by Exception Techniques 
- Operational 



Task A, Program of Annual Review 
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IV. PHASE I ~ PROGRAM REVIEW 

This initial phase of the program has been planned to 
provide the committee with complete familiarity with the Legis- 
lative acts and programs which are the focus of the evaluation 
methods the committee is charged with developing and conducting. 
In this phase the committee will primarily depend upon briefing 
from the State Department of Education. The areas of interest 
to the committee, as stated earlier by the committee, are 
repeated below. 

A. Definition of Program objectives. 

B. Program Operational Procedures. 

C. Methods of determining student population. 

D. Methods of choosing the district and school participants. 

E. Evaluation and measurement methods. 

F. Program content as organized in subcategories. 

G. Program scope in terms of dollars and participants. 

H. Program assessments. 

The committee may request bibliographies providing sources 
to additional information and assistance in obtaining these 
sources . 

The committee may desire and request a synopsis and status 
report for future reference from the proper department personnel. 

The committee will request future status reports on the 
progress of work under these legislative acts on each subsequent 
January and June as regards the work in the preceding term. 

The committee will also request that the State Department 
of Education set up field trips at least twice a year to permit 
the committee to visit the school and to see programs in operation. 
One such visit should be made during this current school term. 

V. PHASE II ~ COMPONSNT STUDIES, 

Task A - Program Study aiid Adoption 

One of the first studies of the committee will be to review , 
modify, and adopt this Program Plsm or an alternative. In its 
May 1970 meeting the committee will have opportunity to be briefed 
by the Legislative Assistant who framed the act establishing this 
committee . 
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The Program Plan and time line presented herein need 
not be considered to be firm even after modification and 
adoption; but should be subject to change as the program 
develops . 

Task B - Fiscal Methods Study 

Task B-1 of this Fiscal Methods Study is to become 
cognizant of the program Planning & Budgeting system recently 
adopted by the State for all State functions. This is again 
a procedure adapted from the Department of Defense. The 
committee has planned an additional familiarization and 
presentation during its May meeting by the Vice Chairman, 
Dr. Agnes Robinson. It is contemplated that the committee 
will request and receive additional presentations on this 
subject during the June meeting. 

The second portion of this task is to learn how the 
various school districts handle their internal finances in 
order to get accurate assessment as to the actual costs of 
these compensatory educational programs. The committee should 
satisfy itself that the costs reported from the various school 
districts are comparable both at present and in the future 
when the PPBS System is fully operative. 

The third portion of this task is to evaluate the ade- 
quacy of the fiscal measuring and reporting system which will 
be utilized for control and measurement of the programs under 
the committee's surveillance. If modifications or changes are 
desirable, the committee will formulate suitable recommendations. 

Task C - Study of Objectives 

As noted in the introduction, it would be highly de- 
sirable from the point of view of cost effectiveness measure- 
ment, if objectives could be expressed for compensatory 
educational programs which could be measured quantitatively. 
The introductory remarks in the Title I Annual Report state 
that "compensatory education is aimed at insuring that every 
child will receive an equal opportunity to succeed to the 
full extent of his potential regardless of his economic, 
social, or cultural background." It is not at all apparent 
how this statement of objective can be reduced to quantitative 
terms. However, it may be possible to break it down into com- 
ponents which better lend themselves to quantitativeness. The 
potential capacity to "achieve" was at one time thought to be 
measured by "intelligence tests." However, these are now 
known to be misleading because they presuppose a knowledge of 
language and society which might not exist. As example, in 
lieu of this, we might think of "potential" in terms of a 
demonstrated rate of absorption, utilization and understanding. 
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A sub-conunittee should be established to study this fun- 
damental task of defining acceptable objectives in measurable 
terms. It may choose to request assistance from the State 
Department of Education and from the academic community. 

It is difficult to perceive of an approach to the measure- 
ment of education which does not involve some kind of testing. 
However, the specific objectives chosen should help to crystalize 
the What, Where, How, and Why of the tested population and of any 
comparison population. However, this plan places the responsi- 
bility for investigating measurement methods in a separate subtask. 

Task D - Study of Methods 

t 

The purpose of this task is to study the state-of-art of 
educational measurement. Initially, the field will be surveyed 
in general; assistance will be drawn from the State Department of 
Education and the academic community. The question of the 
accuracy of testing should be addressed and the factors which 
affect it. What kinds of tests have been developed? Can they be 
categorized? What do they show, relative to accomplishment in 
terms of the acquisition of knowledge, its understanding, and its 
utilization. 

The program is planned so that the sub-committee can learn 
about the subject first, and then synopsize infomation for the 
full coomittee to assist in the other sub-committee's activities. 
Thus, when Task C Committee has reached a conclusion, the Task D 
activity will be in a position to recommend specific methods of 
measuring the attainment of the objective chosen shortly there- 
after. Among other matters, the cub-committee should determine 
if the various programs (i.e.. Title I, Miller-Unruh, etc.) should 
be measured differently. 

Task E - Study of Scales 

Task E has been planned for initiation just prior to the 
completion of Task C. Once the objective has been set, this sub- 
committee should consider the suitable scales by which each 
attainment might be measured. Apparently the State has already 
adopted "equivalent school years attainment per year" as a scale. 
This might be considered along with other approaches. In addition, 
the committee should establish what is "acceptable" or 
"unacceptable" as determined by the scale established. 

The sub-committee should also consider and attempt to 
identify intangible factors which affect its scale and measurements. 

The sub-committee on Task E will work closely with that on 
Task D and strongly influence the rtoomnendations of that sub- 
committee. They will also interrelate with the work of the Task F 
sub-committee. 
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Task F - Intangibles and Equivalents 



This task is to consider possible quality of attainment 
and accomplishment which could result from these compensatory 
educational programs but which are not measurable in the sense 
sought above. This is not to address the effect of environment, 
language difficulties, etc., on the results of the measurement. 
Rather, this task should assess such factors as improvements in 
study habits, change in student attitude, etc. This sub- 
committee will determine how to define these factors and how to 
establish equivalent quantitative terms to factor or operate upon 
the other quantitative measurements in some fashion. 

A further and difficult qualitative assessment to make is to 
introduce into Title III considerations, the probability that 
Title III experimental programs which have been found to be success- 
ful will actually be introduced into the curriculum after Federal 
and State support are no longer available. 

VI. PHASE III -MODELING 

Task A - Expimental Modeling 

The program plan presented assumes that the committee has 
achieved enough background by the Fall of 1970 to undertake study 
of its evaluation model. The approach presented is similar to that 
used earlier in the plan. A sub-committee will evaluate the 
current state-of-art as it exists. They will utilize the State 
Board of Education and the academic community to obtain background 
information and data. They will then utilize this information and 
the prior accomplishments of the preceding phase. In effect, this 
task is an integration of what has gone before. It is envisioned 
that the model chosen could consist of results from one or more 
testing programs integrated by a pattern of mathematical operations 
(i.e., ratios, differences, etc.) plus a procedure for modifying 
the test data, the method, or the final decermination which is re- 
lated to the nontangible factors. It would be hoped that the 
results could be plotted on a definable scale which may be inter- 
preted in degrees of program acceptability. 

Task B - Comparison with Available Data 

The effort envisioned in this task is to collect data from 
ongoing programs which can be utilized in testing the model 
chosen. The objective will be to excunine the kind of results 
which the model produces. It may occur that no data is available 
which might be suitable. In this case, the sub-committee may 
initiate the generation of data. It will utilize the assistance 
of the sub-committee on Task D, below (and will conduct a test of 
both model and procedure simultaneously) . 

Task C - Procedures 

The Task C Committee will establish a recommended procedure 



33 

-15- 



for collecting the data required for the operation of the evalua- 
tion model. Typical problems which might be addressed include: 
Can the same forms and format be employed for all programs? will 
the performing districts be held more rigidly to procedure and 
method? Detailed procedures should be worked out in a preselected 
level of tuiiformity and designed so as to ensure that the data 
required for the evaluation model is obtained and is comparable 
from one program to the next. 

Task D - Procedural Testing 

This will be the final model testing planned. It may be 
desirable to extend the period allowed in the plan or to utilize 
only partial implementation (see below) to be iEollowed by a 
second evaluation period at a later date. This experimental 
procedural test will probably be conducted by the State Depart- 
ment of Education with Committee direction and assistance as will 
all other Cononittee instigated testing. 

VH. PHASE IV - IMPLEMENTATION 

Task A - Implementation Studies 

The purpose of this study is to pinpoint those changes 
necessary in the administration of these laws so as to implement 
the Committee recommended evaluation model. This may introduce 
greater standardization in program management, place changed or 
increased responsibility upon the performing school districts, 
require additional data from them, etc. The Committee may wish 
to recommend a training session for school program administrators 
or a revision to portions of the current guidelines. 

Task B - Formulation of Recommendations 

The Committee will have completed adequate work at this 
point in the plan to make firm recommendations to the State Board 
of Education, the State Department of Education and the Legisla- 
ture. These recommendations should include a full description 
of how the Committee suggests it will conduct program and cost 
effectiveness evaluations of compensatory educational programs. 

Task C - State Board Study emd Acceptance 

The Committee will support the State Department of 
Education during the evaluation of the Committee's recommendations. 
As a result of this study, the Committee may choose to modify 
portions of its recommendations. 

Task D - Implementation Supervision 

Following acceptance of the Committee's recommendations 
by the State, the Committee will assist the State Department of 
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Dd'Vvi iopiUt^r. : of 



Mar.ageiment by Exception Techniques 



It is envisioned ta^z tna size and number of programs 
involved will still prav^-p.t the Coinniittee from evaluating each 
one individually. Therefore, ztie Coromittee should establish a 
plan utilizing mana^e;.-!^;^;. oy .ixception techniques to separate 
those areas requiring ^tten:;ion from those that can be handled 
within the established procedare. 

VIII. PHASE V - OPi' RA T XQN:^... 

The Committee havin.:j oorapleted the critical planning task, 
will then be operat ior.L*! , In -che course of its review, it is 
anticipated that the Cor.miictea will see areas where it may make 
further studies and recoraaendations . Upon successfully negotiati: 
the work noted above, che Conimittee will have adequate background 
to define useful areas for further studies and to provide further 
recommendations both as to curricula and administration. 

IX. STAFFING 

Having prograrun^id tne activity of the Comir.ittee, it is now 
possible to define supporcing c^ssistance requi f^i^i. it is recom- 
mended that the Comro.ittee aire a staff consisting o. one profes- 
sional and one full-time secretary. 

The professional wi.l^, at the requiryst ot the conmittee 
chairman and sub-commi7:cee chairmen: 

a. Definitise x.he requests of the Coiimiv-t*. .. ot the State 
Department of Education* arid coordinate the responftc--^ 

b. Locate ar.d rfccommer.d suitable cor.suJtants in specific 
areas of interest to r.ne Co.Ti.Viitc£e. 

c. Conduct licera-uura surveys at the ciirectiovi of the 
Committee . 

d. Plan Coirirait ;ae fiold trips. 

e. Sunuaarizvi su::.~couv.:ir.tvi.e activities for the use of the 
full Committee. 

f. Prepare repor ct; , 

g. Piovide cdri: t.-'o:-. . :. export as required. 



The professional should be experienced in educational 
programing, methods and administration. Most desirably he 
(or she) should be experienced in compensatory education and 
intimately f«amiliar with appropriate Federal and state statutes. 

The secretary will provide support to the professional, 
prepare and maintain committee records, provide arrangements, 
agenda, etc., as may be required and provide additional 
assistance and support to the committee members upon request. 
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REVIEW OF WORK PLAN - January 1971 



In January 1971, Mr. Ginsburgh reviewed the accomplishments of 
the Committee up to that time and projected the tasks that needed 
to be completed by each of the sub-committees in the development 
of the methodology for assessing cost effectiveness. The tasks 
by sub-committee were as follows : 

Sub-committee on Objectives - The Committee has limited its work 
to the subject areas of reading and mathematics. To be compared, 
progrouns must have the same objectives. The Sub-committee on 
Objectives should compile and adopt a set of detailed instruc- 
tional objectives which are individually measurable and which, 
in total, describe what we mean by reading skills and mathematical 
skills. This will provide comparable, measurable, instructional 
objectives. The identification of all of the measurable instruc- 
tional objectives of interest to the Committee is the task of this 
sub-committee . 

Sub-committee on Measurement - The: choosing or devising of a 
test or set of tests to measure the instructional objectives 
identified by the Sub-committee on Objectives is the task of 
the Sub-committee on Measurement. 

Sub-ccMtimittee on Scaling - The success of any given instructional 
program is strongly dependent on the characteristics of the popu- 
lation taught and the environment in which it is taught. The task 
of the Sub-committee on Scaling is to identify those population 
characteristics and environmental factors that seem to have 
an influence on learning and to divide them into measurcJ^le and 
nonmeasurable varieJsles. 

Sub-committee on Costs - The program costs consist of all cost 
elements identified with the program. The task of this sub- 
committee is to identify the costs that should appropriately be 
charged to a program and a format for collecting these costs. 



REVISION OF WORK PLAN - August 1971 



Following the completion of Phase I and the progress report to 
the State Board of Education, a Sub-conunittee on Planning, under 
the chairmanship of Dr. Levis Kohler, met several times and pro- 
duced several reports pertaining to future action. A staff 
review and compilation of this work produced a sequence of tasks 
for the completion of Phase II, III, IV and V with objectives for 
each phase. The tasks, objectives and time line were as follows: 

In projecting the tasks required to complete Phase II, 
Component Study, the following should be included: 

1. Recommendations for terminal objectives for 

a. Demonstration project, grade 7 

b. Grade levels 1-12 

2. Recommendations for measuring instruments for 

a. Demonstration project, grade 7 

b. Grade levels 1-12 

3. Recommendations for student characteristics and other 
variables to be included in the 

a. Demonstration project 

b. Cost Effectiveness Model 

4. Recommendations for all resource inputs to be included in 
the 

a. Demonstration project 

b. Cost Effectiveness Model 

5. Recommendations for the format in which each of these 
types of data should be collected and reported. 

Phase III, the construction of the Model, would follow Phase II and 
would be primarily the assembling, testing and modifying of the 
outputs of Phase II. 

Phase IV, the implementation of the Model, would require the appli- 
cation of the Model to selected projects of the several funding 
sources named in AB 606 with a determination of any changes required 
in the initial application and in the final reporting of these 
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projects . 



Phase where the Model would be put in general use with the 
appropriate categorical programs, would be the responsibility 
of the State Board of Education. 



Objectives for Phase II 

1.0 By October 1, 1971, the Sub-committee on Objectives will have 
worked with personnel from the Riverside Unified School Dis- 
trict in their formulation of specific behavioral objectives 
for the students participating in the demonstration program 
in reading and math at grade 7. 

2.0 By May 31, 1972, the Sub-committee on Objectives, with the 

assistance of the Department of Education and other agencies 
or consultants as required, will have formulated specific 
behavioral objectives in the content areas of reading and 
mathematics for student achievement levels at the end of 
each grade 1-12 in schools throughout the state. 

3.0 By October 1, 1971, the Sub-committee on Measurement will 

have worked with personnel from the Riverside Unified School 
District in their determination of the measuring instruments 
to be used in assessing the degree of success in attaining 
the objectives formulated for the demonstration program in 
reading and math at grade 7 and the format in which the data 
will be collected. 

4.0 By May 31, 1972, the Sub-committee on Measurement, with the 
assistance of the Department of Education and other agencies 
or consultants as required, will have determined the measuring 
instruments to be used in assessing the degree of success in 
attaining the objective lormulated for students in each 
grade 1-12 in schools throughout the state and the format 
in which the data will be collected. 

5.0 By October 1, 1971, the Sub-committee on Scaling will have 
worked with personnel from Riverside Unified School District 
in their identification of the student characteristics and 
other variables that may be considered in the demonstration 
program in reading and math in grade 7 and the format in 
which the data will be collected. 

6.0 By May 31, 1972, the Sub-committee on Scaling, with the 

assistance of the State Department of Education and other 
agencies or consultants as required, will make recommenda- 
tions as to the student characteristics and other variables 
that may be included in the Model based on experience gained 
with the Riverside Unified School District and other sources 
as appropriate. 
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7.0 By October 1, 1971, the Sub-coiwnittee on Cost and Finance 
will have worked with personnel from the Riverside Unified 
School District in their identification of the resource 
inputs that will be documented in the demonstration program 
in reading and mathematics in grade 7 and the format in 
which the data will be documented. 

8.0 By May 31, 1972, the Sub-committee on Cost and Finance, with 
the assistance of the State Department of Education and other 
agencies or consultants as required, will make recommenda- 
tions as to the resource inputs that will be included in 
the Model based on experience gained with the Riverside 
Unified School District and other sources as appropriate. 



Objective for Phase III 

1.0 By October 1, 1972, the Advisory Committee on Program and 

Cost Effectiveness will have completed a Model by which cost 
effectiveness can be measured in the areas of reading and 
mathematics . 



Objective for Phase IV 

.0 By Marah 1, 1973, the Advisory Committee on Program and 
Cost Effectiveness will have applied the Model to selected 
projects ill Title I ESEA, Title III ESEA, Miller-Unruh Basic 
Reading Act of 1965 and Chapter 106 of the Statutes of 1966, 
and will make recommendations for any changes that may be 
required in the initial application fonrs or the final re- 
porting forms for projects under each of the above listed 
funding sources. 



Objective for Phase V 

.0 By April 1, 1973, the Advisory Committee on Program and 
Cost Effectiveness will recommend to the State Board of 
Education a Model which may be used to assess the cost 
effectiveness of programs in reading and mathematics. 
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r.KPORTS OF THE SUB-COMMITTKES 



As sub-conimitteos made progress towards their goals, reports were 
submitted to the full Committee for consideration. The reports 
that follow have been edited somewhat to minimize redundancy, but 
the complete reports are in the permanent files of the Committee. 



SUB-COMNITTEE ON OBJECTIVES 



Report to full Committee, September 11, 1970. 

Based on the discussion of those present at the sv;b-committee 
meeting on September 10 and 11, a few introductory remarks may 
help to clarify the material developed which is presented in the 
following pages: 

Considerable confusion and controversy arose over the target group 
related to the sub-committee's basic task, the development of ob- 
jectives. Were these to be measurable objectives for the AB 606 
Committee; or, for the several populations, pupils, teachers and 
parents, to be enhanced through Title I and Title III of ESEA and 
the Miller-Unruh Reading Programs? Reference to the program plan 
adopted by the Committee at its July meeting served to clarify 
the issue. The following statement is made on page 9 of this 
document: "... it would be highly desirable from the point of 
view of cost effectiveness measurement, if objectives could be 
expressed for compensatory educational programs which could be 
measured quantitatively." The sub-committee's assignment, there- 
fore, was to provide a base, at least, for a hierarchy of goals 
and objectives for the recipients of the three compensatory 
programs . 

Prudence dictated that the sub-committee begin with those con- 
straints on outputs or products which had been mandated or defined 
through Federal and State enactments, rules and regulations of 
the California State Board of Education, and procedural and 
approval requirements developed by the various, offices concerned 
with these programs in the California State Department of Education. 
In addition, perusal of the source documents enabled the sub- 
committee to analyze the scope of the written intent of the three 
programs. In the sections which follow the specific sources of 
key statements are enclosed in parentheses. 

In order to distinguish between goals of the Committee and goals 
of program recipients, statements pertinent to each were prepared; 
two related to the overarching mandate given to the AB 606 
Committee by the Legislature and three for the target populations 
identified in the published materials related to the separate 
programs. Where explanatory information seemed necessary, as in 
the definition of a specific population, it v^as added and the 
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source citud. 

The framework which twiiows is still incomplete. It requires 
detailed translation into (1) program objectives stated in terms 
of the population to be benefited , and (2) further delineation 
of each program objective into performance objectives which can 
be measured through a system of qualitative and quantitative 
evaluation. For example^ the six program objectives of Title I 
were drawn directly from the McAteer Act - SB 482. Objective 1.1.1 
would be improved by stating it as follows: 



1.1.1 Pupils will read at or above the grade 

equivalent appropriate for their chrono- 
logical age as measured on criterion 
referenced tests and on a standardized 
test of reading achievement. 



More specific performance objectives for pupils derived from this 
program objective follow: 



l.l.lul Demonstrate increasing mastery of the 

reading vocabulary through word memory ^ 
word recognition skills and context clues. 

1.1.1.2 Read with comprehension and apply what is 
read in a meaningful and productive manner. 

1.1.1.3 Read for appreciation^ enjoyment and to 
obtain information. 

1.1.1.4 Interpret printed material effectively ^ 
make inferences and draw reasonable 
conclusions . 



1.1.1.5 Demonstrate the use of reference skills. 



This appears to be an appropriate task for the Committee's profes- 
sional consultant in cooperation with representatives of the 
California State Department of Education, 



Dr. Agnes S. Robinson^ Chairman 

Mr. Morris E. Currey 

Dr. John A. Geddes 

Dr. Lewis T. Kohler 



SUB-COMMITTEE ON MEASUREMENT 



The minutes of the sub-committe meeting of November 20, 1970 
cover the primary concerns in the area of measurement. 

Assistant Chancellor Elwin V. Svenson and Professor Garth Sorenson, 
both of UCLA, testified before the sub-committee and their com- 
ments are summarized in the minutes as follows: 

Chancellor Svenson began his presentation by stating that he felt 
the Advisory Committee on Program and Cost Effectiveness was very 
important to the whole state of affairs in education in the country, 
particularly in California. He told the Committee that they ought 
to be aware of the validity of tests, the variety of tests and how 
they are being used. The main concern in education today is ele- 
mentary education and the focus of interest ought to be at that 
level. 

Utilization of tests are for (1) diagnostic purposes , and (2) for 
eva l uating the end results . The first use of tests has to be 
related to the individual for the purpose of individualizing 
instruction. In other words, the diagnosis must be related to the 
things that can be taught (to skill areas) , and should describe 
the program the student is in. The second use is to check the 
progress of the student with his program of study, i.e., to 
evaluate the effectiveness of the program. 

Next comes the task of reporting the results to the general public, 
school boards, teachers and the children. Chancellor Svenson felt 
that most people do not understand the tests enough to understand 
what they actually report. He told the Committee that they ought 
to find out how tests are used before a report could be submitted 
to the Legislature. 

The main concern should be with the system of testing and manage- 
ment which can be expected to be utilized at the classroom level. 
Then the test can be interpreted to people by means of standards 
and norms, to tell them what has been going on. Definitions of 
programs must be understood by the school personnel, general 
public, parents and children before you can talk about cost 
effectiveness. 

Because personnel costs dominate instructional budgets, utiliza- 
tion of the teacher's time should be a concern of the Committee. 
They should look at schools and see how teachers are using their 
time and make comparisons with other schools. The question is, 
if some schools are spending more time, are they getting more 
results? An individualized approach presents a better under- 
standing of what is going on. A study, focused on specific 
sub-populations should be made of schools using the diagostic- 
prescriptive technique as compared with traditional schools. 
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Dr. Garth Sorenson (UCLA) spoke on updating concepts in testing. 
One of the biggest changes he noted was the way in which tests 
are being used. In the past, tests were for measuring children, 
but now they are also used to evaluate instructional programs. 
Schools can be compared to other enterprises because they have 
processes and products (product is children's learning). The 
problem with this is results cannot be observed directly, but you 
can observe consequences — change in performance, i.e., product 
IS defined as change in pupil performance. 

The major purpose of the educational process may be to change 
student performance on an intellectual test. We will want to know 
what the student can do at the beginning of a course. Using the 
dxagnostic approach, some things the teacher should know at the 
beginning of a course are the student's skills, habits, attitudes 
and a work sample (measuring accumulated achievement). The second 
step IS to give the student some assignments designed to move him 
from where he is to where we want him to go. The third step is to 
test to see if they are working, i.e., to see if the teaching pro- 
cedures are making a difference in the student. Interesting tests 
may be developed which compare an individual's present performance 
with his past rather than with others. 

A question was raised at this point as to whether or not this 
lipplied that it would be desirable to develop tests that are not 
now available. 

Dr. Sorenson replied that rather than tests, the problem is to 
define kinds of instruction, i.e., to decide what kind of instruc- 
tional procedures are needed. Most teachers don't do this, but if 
they did, it would tell us more what we are doing. He went on to 
state: (1) We need to develop tests that are interesting to 
children; (2) Make sure they're not too easy or too hard; (3) 
Instead of comparing performance to another student, compare it to 
each individual's past performance. In other words, the instruc- 
tional process should be redesigned instead of the testing process. 

Dr. Sorenson spoke further about reinforcement in testing. First 
of all, the student should take the test; then the teacher should 
get feedback from this test. Special assignments should then be 
suggested and the student should be allowed to take the test again 
whenever he feels ready. 

The notion of standardization is important but it should be based 
on criterion referenced tests. You have to know what you are 
measuring and what the results indicate. Standardization means 
the ways of giving a test and scoring it. In other words, it means 
two people scoring a test and coming up with the same score. Tests 
are used to appraise students and to show what learning has occurred. 

In summary. Professor Sorenson stated that the test data should be 
used not to evaluate the children or the teacher, but to evaluate 
the method of instruction. 
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A question was raised by Dr. Geddes as to whether or not these 
tests could be used to provide the Legislature with some account- 
ability of what is going on in the classroom. Professor Sorenson 
replied that this was possible because a test is a work sample 
of skills. Goals should be set up and students given a certain 
type of test to see wliere they are at the beginning of the year. 
At the end of the year the test should be repeated to see what 
changes have occurred. 

A question was raised concerning the possibility of uniformity 
for a testing program for this Committee. Professor Sorenson 
replied that he didn't feel that a uniform test program is possible; 
but, policies regarding nature of testing programs can be uniform 
anc? the way in which tests are used and reported can be consistent. 

A question was raised as to whether or not the Committee should 
compare different programs to compare outputs and cost; i.e., in 
comparing program A and B, what if the populations have different 
social and economic composition? How can judgment be made as to 
value of each program? 

Professor Sorenson replied that in theory you could try A and B 
in several kinds of schools, but in practice, it is difficult 
because of value structures and political implications. 

The question was asked, "Do you think standardized tests, as 
presently used, are sensitive to teaching strategy (process 
measures)?" Professor Sorenson 's reply was negative because the 
tests were not built for measuring what teachers are trying to do 
in the classroom. 

Components that he felt must be considered in a cost effectiveness 
model are: (1) attitude of principal; (2) attitude of staff and 
facility regarding teaching; (3) whether or not thu school allows 
time so teachers can utilize information to modify curriculum; 
(4) parental involvement, and (5) flexibility betwfjen school and 
central district. 

You can come up with criteria by comparing: (1) principals - 
their attitudes in relationship to students, teachers and community; 
(2) attitu*^?* of faculty - their belief in the diagnostic approach, 
and (3) parents - their understanding of what is going on in the 
school. 

Professor Sorenson feels that we need repeatable programs; further, 
what repeatable programs we have should be investigated as to their 
costs and the number of personnel required to administer programs. 

When questioned regarding the use of indirect measures of effec- 
tiveness. Professor Sorenson told the Committee that they should 
not evaluate a whole program because it is too complex, but rather, 
they should evaluate a program, a piece at a time. 
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Professor Sorenson was asked to restate his interpretation of 
formative evaluation. He stated that the purpose of formative 
evaluation was to find out if a program is working or not; and 
secondly, the task of getting information that you need to improve 
the program. The second step was emphasized as being more im- 
portant than the first , 
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Mr. 
Mr. 



John A. Geddes, Chairman 



Allen S. Ginsburgh 
Lewis T. Kohler 



Clarence R. Newby 
Alvin L. Rosenthal 
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SUB- COMMITTEE ON SCALING 



Report to full Conur.ittee, January 17, 19 71. 

As a discussion starter, Dr. McCormick sumraarized the groupings 
of schools used in connection with the Title I, ESEA third grade 
study pointing out the various characteristics upon which the 
groupings are based (ethnic characteristics, degree of poverty 
and size of school) . 

Several papers provided by Dr. McCormick were used. 

(1) 2 SES predictors of achievement are family income and race^ 

(2) Other things to be considered in "scaling" 

(a) Reviewed Levin's paper which contained the following 
variables along with regression weights2 

Negro student's verbal score 
Reading material in home 
Siblings (positive equals few) 
Parents ' education 
Science-lab facilities 

Teacher salary (in thousands of dollars) 
Teacher verbal score 

(b) Kiesling's paper used the following variables3 

Index of occupation 
Teacher-pupil ratio 

Per pupil expenditure on books and supplies 

Teacher salary top decile 

Value of school property per pupil 

Per pupil expenditure on principals and supervisors 



1. Burke, Kelley and Gams, Educational Programs for the Culturally 

Deprived - Need and Cost Differentials , National 
Educationa* Finance Project, Special Study No. 3, State 
University of New York (Albany) 1970 

2. Henry M. Levin, Cost Effectiveness Analysis and Educational 

Policy - Profusion, Confusion, Promise , Dec. 1968, 
Stanford Center for Research and Development in teaching. 

3. Herbert J. Kiesling, The Relationship of School Inputs to Public 

School Performance in N.Y. State , October 1969 (Consultant 
to RAND Corp. ) 

Using the preceding variables, he did a separate analysis 
for each of the following socio-economic groups: 
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1. Professional persons 

2. Proprietors, managers, officials 

3. Clerks and kindred workers 

4. and 5. Skilled and semi-skilled workers 

6. Unskilled workers and servants 



(3) The Committee considered the demand for operational approaches 
to cost effectiveness analysis^ 

Three major stages: 



1. Identifying alternative programs 

2. Estimating the cost of each 

3. Estimating the effectiveness of each 



A fourth stage, which may be the most important, frequently 
results : 



4. Designing a new program that is better than any previously 
considered. 



Discussion by the sub-committee resulted in agreement that the 
immediate task would be to identify "pupil variables affecting pupil 
achievement that are readily measurable and for which it is possible 
for school districts to provide data." The following ten variables 
were identified and stated in tentative form: . 



1. Family income. 

2. Ability to communicate orally in English. (It appeared 
that the problem involved is really a handicap in using 
English in the school situation. Such a handicap might 
appear among pupils of any ethnic background. It seemed 
that use of the actual language handicap as a factor 
rather than the ethnic background is most realistic.) 



3. Academic ability or "intelligence." 

4. Sex. 



5. Educational level of parents. 

6. Sensory handicaps. 

7. Migrancy or mobility. 

8. Achievement level. 



9. Family size. 
10. Both parents in the home. 
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4. Carpenter, Haggart, Levien, Rapp, Root and Sumner, Cost 
Effectiveness Analysis , Staff paper of the RAND 
Corporation. 

As an exercise in determining the possibility of assigning dif- 
ferent values to these factors, the attempt was made to place them 
in three value categories. With rather general agreement in most 
cases, the values assigned were: 

Group 1" 1, 2, 3 and 5; Group 2 - 6, 7 and 8; Group 3 - 4, 9 

and 10. 

It was recognized that perhaps other variables should be added to 
this list and that it is necessary to determine what research results 
are available that would help with the development of such a listing. 
Dr. Mccormick was asked to provide the sub-committee with information 
as to available data of this type at the next meeting. 

The question of "program" variables, or descriptors, was varied; i.e., 
schools differ in the approaches used for instruction in compensatory 
reading programs. Whether this is a "scaling" problem was not resolved. 



Mr. Donald E. Kitch, Chairman 
Dr. John A. Geddes 
Dr. Agnes S. Robinson 
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OTHER CONSIDERATIONS IN SCALING 



As the Sub-committee on Scaling investigated the area of student 
characteristics that related to the student's ability to profit 
from instruction, it became evident that there were other factors 
that also influenced the student's learning rate that needed to be 
considered. There was also a lack of agreement as to what should 
be included when describing a program for replication, or what 
program elements needed to bo the same in order to say you have 
the same program in two or more schools. Items that were discussed 
were scope and sequence of content, teaching strategies, supplies 
and equipment, staffing patterns, staff characteristics, funding, 
class size, physical facilities, administrative and support services 
and many others. 

Contributions to the thinking in this area were heard by the Com- 
mittee and a summary of some of them is presented here. 

Mr. Edward Gustafson, Chief of the Bureau of Program Planning and 
Development and Administrator of Title III discussed his graph on 
instructional elements. Elements of this graph are; Instructional 
setting, size of group, library resource area, multi-media center, 
laboratory and community activities. Variables are; (1) learning 
style, (2) background, (3) entering capability, (4) motivation, and 
(5) physical ability. Reading content was classified as (1) decoding 
(work skills, etc.), (2) meaning, (3) comprehension, (4) appreciation, 
and (5) skill use. 

To gain information regarding effectiveness from any instructional 
program, we need; 

I. A set of rules for selecting the tests. Criterion 
referenced tests measure the intellectual skills the 
teacher is trying to teach; they will be used in; 

A. Measuring what the student knows at the 
beginning of the course. 

0 

B. To check the effectiveness of individual lessons. 

C. To check what student knows at the end of the course. 

II. A set of repeatable instructional procedures designed to 
improve the student's performance on criterion referenced 
tests. These will have to provide for differences among 
pupils in learning speed and motivation. Will need some 
remedial routines. 

III. A set of formative evaluation procedures foi" testing and 
improving the constructional tools. Will include: 
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A. A record of each student's progress. 

B. Monitoring procedures to see that the teacher used 
instructional program correctly. 

C. Comparison groups. 

Part III must be built in at the beginning of the program. 

Dr. Rider discussed program variables that should be considered in 
a cost effectiveness model. 

•Some of the most important program variables he covered were: 

1. Criteria for selection of students. 

2. Diagnostic systems. 

3. Organizational system. 

4. Frequency and grouping for instruction. 

5. Hours of instruction (pro rate time of instructional personnel 
and get instructional cost schedule) . 

6. Instructional techniques (80% of cost is personnel). 

7. Behavior modification techniques. 

8. Hardware, software systems. 

9. Communication network. 

10. Instructional personnel characteristics - quality factor. 

11. Staff developing (inservice training) . 

12. Management systems (planning, monitoring the system, lines of 
decision making - authority) . 

13. Program costs must include the district costs and supplemental 
costs . 

A paper prepared by Ruth Overfield analyzed factors associated with 
varying degrees of program success. As input or predictor variables, 
she would include : 

1. Number of specialist reading teachers. 

2. Library facilities and personnel. 

3. Transiency of students (staff?). 
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4. Number of students who come from non-English speaking families 
or where other than English is the primary language at home. 

5. Class size. 

6. Teacher salaries. 

7. Duration of instruction. 

8. Tax rate - assessed evaluation per ADA. 
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SUB -COMMITTEE ON COSTS 



The work of the Sub-committGe on Costs , prior to the Riverside 
project, was oriented towards describing the existing situation 
and identifying practices that would need to be changed in order 
to gather the kinds of data required in a cost effectiveness 
assessment. At the meeting on May 17, 19 71, Mr. Newby, 
Dr. Lindman and Mr. Rosenthal presented some of their views. 

Mr. Newby commented on school accounting procedures which has been 
summarized as follows: 

Present accounting systems in school districts are the result of 
recommendations and items involved in the Education Code. The 
State suggests what is to be done as an audit procedure. The entire 
problem is state controlled, or state oriented. All expenditures 
are line classified - administration, instruction, health services, 
operation of plant, maintenance of plant, fixed charges, pupil 
transportation, food services, community services, capital outlay, 
etc. 

Generally, a clerical person is charged with the matter of accounting 
for the cost of a particular project. This is done not by a formal 
set of books, but by a series of files where purchase orders and 
invoices are accumulated, and if fortunate, you will have a work- 
sheet set out which tends to characterize expenditures and compare 
it with the budget. We are yet to find this file complete. The 
responsibility is in the hands of a clerk or person who does this 
in their spare time. A more effective and accurate set of records 
should be made. No attempt is made to supply costs in detail. A 
certain amount is allocated to equipment, supplies, salaries and 
every effort is made on the part of the school district to expend 
these funds. At no time can we find out if there is an evaluation 
going on, when decisions are made concerning projected materials 
and equipment; it really is not necessary. There is no detailed 
costing as I can follow it, intricate costs - classroom costs and 
some of the minute costs that we would be interested in is not 
being done, and indeed apparently not being required. No detailed 
cost analysis and effectiveness is available. In the area of 
accounting the State is going to have to take the leadership role. 
It has t£dcen the role in the past to bring the accounting systems 
to where they are now, but public accounting really hasn't had a 
hand in suggesting or medcing any recommendations in this overall 
cost accounting. Now is the time to offer certain suggestions 
from the accounting profeission. 

Concerning tha PPBS approach, will this approach solve some of 
the problems? If the new system at this point could entertain 
some of the problems that inherently are in this cost approach, if 
this new system can zero in on some of these problem areas, it may 
well be that this system can pick up the bits and pieces that this 



Conunittce noocls and bring along the real thing in the accounting 
system we need for our cost effectiveness. 



Dr. Lindman's presentation as summarized: 

Wnat are the problems? We must consider what cost information is 
needed in relation to them. There are three types of analyses: 
(1) Incremental service analyses; (2) Alternate route; (3) Alter- 
nate goals . 

Cost is much simpler than the benefit part. Teacher salary is 
fixed by the school board or State. The salary schedule is com- 
pletely beyond the control of the director who is responsible for 
cost effectiveness of education. In the cost analysis of teaching 
a course, the biggest item is teacher's salary. 

Class size is another factor which determines per student cost. 
It is wiser to base costs on optimum class size rather than on 
actual current class size. The two major factors to influence 
the cost of a particular program is teacher position and salary 
and class size. 

We can make benefit cost analyses in education provided we don't 
put a dollar value on the benefits. If you try to convert the 
benefits to dollars, you will have greater difficulty and lack of 
progress. You must describe and evaluate the service and the 
amount of sarvice rendered. Benefit costs from the economists' 
point of view tend to assume its dollar value on the benefit. 

New program structure under development of PPBS will go a long 
way in identifying program cocts, which are not identifiable 
under th^j old system; getting an accounting system more program 
oriented. This new system should go a long way in aswering your 
questions in the years ahead. Purpose of cost analyses: First, 
be sure you know what you want it for. 

We are charged with dealing or finding out some information about 
costs in specific programs. As individuals at the State and 
Federal level, we have ideas about this program - thct it's 
going to great and wonderous things and indeed the program was 
planned to do those sort of things. But I get a different feeling, 
however, from being around a school. It's a different concept 
entirely. At the school, in the accounting office and the 
business manager's office the prograun is just another way to get 
a few extra dollars in that school because they need the money so 
badly. Problems arise because they'll do almost anything to get 
another $10 - 500,000 in that school. They'll submit a program 
for approval to get that money that has only a vague relation to 
what the program was intended to do; they'll falsify a report, 
because getting the dollar in the budget comes first. This is the 
attitude I sense at most schools. Once the federal prograuns are 
adopted, cost wise the program becomes an orphan. The schools are 
doing largely what they've always done, except when it comes time 
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to report what they have done with the money. My point is that 
every school is different, and yet most of them are the same in 
this regard. They're different in that they have different people 
working to draw together this report, they have different costs 
and programs and projects, but the ultimate requirement in all of 
them is complete the report to see that the money is not taken back 
now that it is spent. 

Another problem encountered in the school districts, and I'm cer- 
tainly sympathetic to this, is that there is a resistance to 
change. There may be changes at the State level, and the adminis- 
trative level, but down at the bookkeeping levels you haven't 
affected a change. That will be the same six months or a year 
from now. 

Another problem that we're faced with in schools, as in industry, 
is a papermill. Everything we do generates another piece of 
paper, and the problems of paper are almost greater than the 
problems of cost. There is a tremendous resistance, even at the 
higher levels, to change when it comes to dealing with paper, and 
it is related directly to accounting systems and costs. 

What is involved in trying to get school costs, dealing with cost 
systems? Budget preparation - How many teachers, name and number, 
classroom or principal, what school are they assigned to. This 
represents 80-85 percent of school costs. If it costs $40 million 
to run a school, at least $32 million is in salaries. The balance 
of the money is spent in supplies and capital outlay. 

The biggest single cost is manpower, representing 80-85 percent of 
the budget; the other 15 percent is immaterial. 

There are three cost systems; (1) Standard cost system; (2) Direct 
cost system; (3) Process cost system. 

The final report of this sub-committee was a joint effort with 
Mr. Walter Parks and members of his staff from the Riverside 
Unified School District. This report is a system developed specif- 
ically for the Riverside project to identify and collect data on 
costs down to elements within a classroom. The report is included 
in its entirety in the Riverside project. 



Mr. 
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Clarence R. Newby, Chairman 
Allen S. Ginsburgh 
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At the February 1971 meeting of the Advisory 
Committee on Program and Cost Effectiveness, 
the Committee made a report of their progress 
and findings to members of the State Depart- 
ment of Education. Mr. Alvin Rosenthal, 
Committee Chairman, introduced the Committee 
and described the charge to the Committee from 
AB 606 and AB 1923. He also described the 
organizational relationships and general 
policies and procedures of the Committee. 
Mr. Allen Ginsburgh summarized the Committee's 
technical progress in the development of the 
methodology and was followed by reports from 
Sub-committee Chairmen Dr. Agnes Robinson, 
Mr. Donald Kitch and Dr. John Geddes. The 
report on technical progress and the sub- 
committee reports are reproduced here in full. 
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oUb-COMMITTji 



£ REPORT ON OBJECTIVLS 



ViiE JOB AHEAD 



Dr. Agnes Robinson 



I do not have a prepared outline or statement for the committee 
and those present today because I did not feel I could be the 
prognosticator without some i .put from the committee itself. There 
was no opportunity to provide that until our meeting last night, 
and I did want to be sure that I was speaking for the committee as 
a whole. 

First of all, the committee recognizes how formidable a task it 
nas been charged with, but it also recognizes that it is just one 
element in a team of five committees and commissions which must 
interface, which must cooperate, which must move forward with a 
common purpose if the goals set by the legislature for these five 
bodies are to be achieved. They are the two commissions related 
to Title I and Title III, the Advisory Committee on Goals and 
Evaluation, the PPBS Commission which Dr. Waters is representing 
this morning, and the AB 606 Committee. I think the legislative 
goals set for this broad, representative team of five include, 
first of all, to establish priorities for education in this state; 
second, to find ways of moving children toward those goals, and 
certainly in a more effective fashion than this state has demonstrated 
since it has had mandated statewide testing; and third, to accomplish 
this through alternate programs that provide m'lximum output at least 
cost. 

In the work of the committee to date, we have uncovered more 
problems than solutions or answers; and to me, this indicates that 
we are moving ahead in expected fashion. At least we are recognizing 
some of the problems that do exist. Many of these tread on very 
sensitive educational toes, but I think the time is past when we 
can afford such sensitivity because of the growing credibility gap 
about education, not only with respect to the public at large, but 
with the Legislature and Governor of this state as well. The time 
has come to put the cards on the table and then play the game as 
tne cards lay. My comments will attempt to do just that. 

The first task that the committee undertook was to set goals and 
objectives for itself and then to determine what the program ob- 
jectives should be for reading and mathematics. The Bureau of 
Program Development in Compensatory Education has made a start on 
this by setting some rather broad profiles for both reading and 
mathematics through program objectives which are much broader' 
based than performance or behavioral type objectives. I think we 
have to agree that the objectives for reading for a child in the 
Miller-Unruh Program are not essentially different than those for 
a child in the Title I program, or in a Title III program, or in a 
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regular school program. Means to achieve those objectives may, 
however, be quite different. 



Thoro are certain basic standards for reading which we expect 
children to accomplish and these must be identified. What are they? 
It might appear logical in terms of expenditures to say they are 
detailed for us in the state reading adoptions and the state mathe- 
matics adoptions. If they are not there, then this state is cer- 
tainly wasting a tremendous amount of money on basal readers and 
other kinds of textbooks for grades 1 through 8. So the com- 
mittee, with the help of the Department and other resource people, 
must identify what these broad program objectives are, on a graded 
basis, for these two subject matter areas. 

The second task is to find that instrument which has the highest 
curricular or content validity for the program objectives in 
reading and mathematics. I say "that instrument" because I 
don't see how we can compare the results of pupil achievement 
as measured on 50 different standardized tests. Recognizing 
all of the inherent weaknesses of norm-referenced standardized 
tests, trying to compare the raw scores and derived scores, such 
as grade equivalents, of so many instruments becomes rather 
ridiculous. Identifying such a test as an output measure in no 
way precludes the autonomy of any district, or school, or teacher 
to select those strategies, including formative ongoing evaluation 
and diagnostic-prescriptive teaching, that will move the child 
toward the attainment of the objectives. But, it does provide a 
common denominator against which we can measure pupil progress and 
then compare that with costs. 

The scaling of the pupil variables, or the weighting, has also 
been considered. I, too, had considerable difficulty in learning 
to interpret Mr. Ginsburgh's definition of scaling, but I made it, 
and I am sure you can too. In the scaling or weighting of these 
pupil variables, a start has certainly been made; as a result we 
really don't have to do a lot of "blue-skying," since there's a 
tremendous body of research already available in terms of those 
variables inherent in a pupil population which can be used as 
predictors of achievement. 

The model Bill referred ^ this morning, which it was not possible 
to apply because of some o£ the data required, such as family 
income, can, I believe, be^arrived at in other and equally valid 
ways. The Scaling Committee discussed this yesterday, and 
Mr. Kitch will be reporting more on this, this afternoon. Asking 
about family income may be an invasion of privacy and probably is 
an item which could not be generated on a pupil population other 
than in some gross way, such as the number of families on A.F.D.C. 
It would certainly be highly possible, however, to obtain informa- 
tion on the occupation of the principal wage earner of the family 
or the highest level of schooling achieved. Both of these would 
have high correlations with family income. 
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Armed with these three kinds of decisions, I think the committee 
will then be prepared to look at alternate programs, alternate 
ways of maximizing the effectiveness of the instructional program 
as measured in the pupil product. To do this, we will have to 
satisfy many data requirements, most or all of which will have to be 
derived with the assistance and cooperation of the State Department 
of Education. Initially, perhaps a small pupil population will 
enable us to do some dry runs in attempting to validate whatever 
model is developed, then expanding to a larger pupil population. 
This is similar to what the PPBS Commission has done in beginning 
with six pilot districts and then expanding to fifteen. 

The committee must be made aware of what it is feasible to request 
districts to provide, and I think Jim Waters has had lots of ex- 
perience with this; such as, what inputs is it realistic to expect 
a district to generate? Because if these are unrealistic, we 
should exclude them from the model right at the beginning. 

The committee must also know what kind of computer applications 
are available to it, since the volume of data that we are referring 
to will have to be programmed and run through data processing 
equipment. 

The resources that the committee requires will, perhaps, be dis- 
cussed at greater length by our chairman, Mr. Rosenthal. I would 
certainly appreciate some feedback from those present at this 
meeting as to the probability of the availability of some of 
the data and machine requirements that we are anticipating at this 
point in time. 
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SUB-COMMITTEE llEPORT ON SCALING 



Mr. Donald Kitch 



I've tried to make a brief cumulative progress report. May I say, 
first of all, that I'm glad Agnes confessed this morning that we 
were a little bit bothered by the use of the term "scaling," as 
Dr. Law indicated here this morning. But Al was patient with us 
and after a litttle indoctrination and schooling we, I think, 
caught the concept that he had in mind, and it does translate into 
the term that we commonly use - "weighting." We've had four 
meetings of our sub-committee which includes Dr. Geddes, 
Ur. Robinson and myself. We've been fortunate in having present 
at all of the. meetings other members of the committee because sub- 
committee meetings are always open to other members of the committee. 
This has been very comforting to us as we've wandered around in 
this wilderness. 

We met in Sacramento, San Diego, Los Angeles, and yesterday after- 
noon (again) in Sacramento. At our first meeting, we attempted to 
pump some of the Department of Education staff members who have 
nad experiences with programs where they had to face up to the 
same problems that we were trying to cope with; i.e., identifying 
some of these variables that Al pointed out in his report this 
morning as being significant. There are variables that the school 
has to accept, as conditioners of the learning experience that 
youngsters would have in any school conducted program. 

As a result of these four sessions, we've come out with certain 
tentative results; i.e., certain recommendations and one or two 
comments which represent opinions or feelings. Whatever you want 
to call them, I'll attempt to summarize them briefly. 

The first thing we set out to do really when we got down to work 
was to try to identify certain variables, psychological, socio- 
Ivogical or economic in nature that would fit into this category 
of vari-ibles which the school had to accept and about which it 
really could not do too much (to start with anyhow). Al's report 
this morning included the list of ten (10) that we've put into 
our list, and I'm not going to recapitulate because I doubt if 
it's necessary. We recognize that this is not the kind of list 
that we would want to live with; this is the kind of list that 
we're starting with. We realize that maybe 10 variables of this 
type are too many and we need to do something about chopping the 
list down; combining perhaps, or deciding which of the variables are 
the ones that prove most useful in terms of what has already been 
done in various projects and various experimental activities (as 
a kind of exercise) . 

The next thing v/e tried to do was to see if we could approach the 




problem of wtiiqhting these variables simply by classifying them 
into three categories. Category I being the one to which we would 
assign the greatest weight. Category II the next, and Category III 
the least weight. We found chat we were able, just on a judgment 
basis, you understand, to make these categorizations without too 
much trouble. There was a little argument and, because this was 
a kind of exercise, I'm not going to try to tell you which variables 
fell into which category (we realize that this is just thinking) . 

The second step that we have taken is to raise the question as to 
whether or not it is the responsibility of our sub-committee to 
develop variables that are program related (program components 
for instance) or other varicibles that might be related to the kind 
of a program that is being conducted. Now we have not done this 
job, but we do have certain raw material that has been given to us 
in Committee sessions that could form the basis for beginning to 
approach this job. 

I'd like to say that at our first meetinr the input that we had 
was helpful to us in getting started in our thinking came from 
Dr. Rider; he talked with us particularly about the third grade 
project. Kelly Acosta who talked with us about the experiences 
that they've had in the Miller-Unruh program and Mr. Gustafson 
represented Title III. Dr. Waters was also at that first meeting 
and although we got a little conversation out of him, we had no 
formal presentation at that time; that's one reason why he's here: 
today. 

A third thing we'd like to have clearly understood is (as this 
kind of worried members of our Committee) we want to make it clear 
that as a sub-committee, we want to make plain the fact that we 
don't want any of the activities of this Committee to imply that 
anything we do is intended to exclude in any way the active 
encouragement of experimental programs (that is, programs which 
might not be tailored along the lines that we happen to be think- 
ing) . In other words, we want left clear this area of expeiiroer,- 
tation so that everybody understands that we're all in favor of 
experimental type programs, even if they '^on't fit our formulas 
or whatever we eventually come up with. 

Another thing that became quite evident to us as we went along 
is the problem of data availability. This has been talked about 
previously and so I'm not going to say very much about it. I 
would like to say one thing though - we've been thinking about 
data availability in terms of individual pupils, not in terms of 
classes or schools or school districts, and we recognize that 
this is kind of a "toughy" in terms of whether it's available or 
could be made available even in a group of cooperating experi- 
mental schools. Also it's a "toughy" in terms of what are you 
going to do with it when you get a basketful of it. There's only 
one answer to that, seemingly, and that's computer services 
which we don't have now and which we may or may not have in the 
future. 
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A fifth item that we've given some thought to is the problem of 
having to translate these identified varial los and their weights 
into some form that could be incorporated into models. When you 
look at the list of ten (if we decided to stick by that list of 
ten) , those variables would not lend themselves to ri common 
treatment. They range all the way from sex, where you'd have a 
bi-multiple distribution probably, to other variables that would 
have to be treated in an entirely different fashion. So this is 
a problem that we haven't really dug into yet. 

Then a sixth and last item, representing an opinion of the chair- 
man of this sub-committee - so I'm speaking for myself at this 
point. And, it may be that the other members will turn thumbs 
down on what I'm about to say, but I don't think so. I think 
our sub-committee has reached the point where with another 
working meeting or so, we would be ready to frame, at least in 
general form and maybe rather specific form, the kind of a work 
assignment that would have to be undertaken by somebody who could 
devote full time for a period of six months or so; e.g., to evalu- 
ate the research that has been conducted so that they can tell us 
which of these ten variables that we are playing with now are really 
significant, or are likely to be truly significant; and can back up 
their recommendation with something based on experimentation. 
There is a task that needs 'to be undertaken. 

We need some expert help with this problem of trying to scale 
each of the variables; and I don't see how members of our sub- 
committee are ever going to have time to really any more than just 
outline this job. Somehow or another, we're going to have to find 
some way to get some service, either through a contractual arrange- 
ment or some other way, maybe free service. Lou down here thinks 
he's a pretty good recruiter of free services. 

(Dr. Lewis Kohler interrputed to say, "I don't say that I can get 
people for nothing. I don't know of anyone who will work for 
nothing. They work for cash.") 

Yes, but your interpretation of nothing is a little different from 
zero dollars. Well, that concludes my report, Mr. Chairman, 
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SUB-COMMITTEE REPORT ON MEASUl^MENT 



Dr. John Geddes 



Firsts we've reviewed a fair amount of information concerning 
measurement and Dr. McCormick has prepared a statement that is 
brief and I think rather summarizes our information to date* 
There are two positions taken when educators consider the assess- 
ment of learning performance (this is very brief - very summary 
and I'm sure leaves out a wealth of information, but nevertheless 
as d general statement, I think it's significant). 

One position is that the domain to be tested is very large, 
consisting of knowledge about a vast amount of materials and 
that assessment consists of a sampling from this domain. Tra-- 
ditional performance testing is based upon this premise and 
allows a rating of individuals or groups. These normative tests 
do not sample how effective a program achieves its specific 
objectives; for example, a pullout remedial reading program. 

Criterion-referenced tests have a local educational program in 
mind; the items are matched to the specific objectives of the 
instructional staff and thus they provide feedback for modifica- 
tion of the local program. As the demand for increased effective- 
ness is growing, the development of criterion-referenced test 
items, that measure specific skills, is also increasing. It is 
generally accepted that criterion-referenced tests are of use in 
efforts to individualize instruction. 

Although criterion-referenced tests are in a developmental stage, 
there are a number of places throughout the country that are working 
in this area (one is the Center for Evaluation at UCLA) . There are 
problems in attempting to use criterion-referenced tests for assess- 
ing the outcomes of educational progress. The contents and the 
order of presenting the learning units are not uniform across 
schools. In fact, the trend is toward program development at an 
individual school level. In addition, programs are modified 
periodically during the school year. 

Secondly, the Measurement Sub-committee is planning: 

1. The continuing broadening of the total committee's 
understanding of educational measurement. 

2. To invite specialists working in the area of the 
evaluation of educational programs, to consider how 
best to measure pupil variables affecting pupil 
achievement as identified in Mr. Kitch's Scaling 
report. (Also, it was in the report that 

Mr, Ginsburgh gave.) 

3. To invite specialists from UCLA and other institutions 



O -59- _ 

ERLC 79 



to survey and report on the state of the art of 
measurement of program effectiveness. I would 
certainly hope that we may also draw upon the re- 
sources of the Department in this and hopefully, 
unite certain people in the Department with cer- 
tain people in the top educational institutions in 
the United States , all of whom are working on this 
area. I think that would be a very profitable 
objective. 

4. To determine State Department of Education progress 
on the measurement of progreun effectiveness and to 
develop a closer planning and working relationship 
that would be representative of the State Department 
of Education. 

5. To involve certain schools early in the planning of 
what measurement techniques are reasonsOjle to utilize 
in our schools. As we begin to work into the area of 
model building, where we're beginning to see what 
kinds of measurements we should put into this appraisal 
process, the measurement techniques should go very 
early into a few individual schools to make sure 

that what we are building is really practical and work- 
able; so that we produce something which already has 
had a parallel testing at an operational level in the 
schools . 



Thirdly, we place our highest priority in the following two endeavors 
which we expect to carry out in parallel. 

1. The first is to survey and report on the state of the 
art of measurement of program effectiveness. Assistant 
Chancellor Svenson of UCLA has expressed an interest in 
helping us to do this. 

2. In addition, we wish to invite the State Department of 
Education to share with this committee their progress 
and problems in attempting to measure program 
effectiveness . 
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REPORT TO THE CALIFORNIA STATE BOARD OF EDUCATION 



With the completion of Phase I of the work plan, 
the Advisory Committee on Program and Cost 
Effectiveness prepared a report of their progress 
and findings to the State Board of Education. 
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STATE OF CALIFORNIA 



DEPARTMENT OF EDUCATION 



STATC EDUCATION BUILDING, 721 CAPITOL MAR, SACRAMENTO 9S8U 



May 25, 1971 



The Honorable Wilson C. Riles 



Superintendent of Public Instruction 
California State Department of Education 
721 Capitol Mall 
S'-cranento, California 95814 

Dear Dr. Riles: 

I am pleased to submit to you a report from the Advisory Coomlttee 
on Program and Cost Effectiveness as required by the Education Code» 
Section 6499.203. 

This report reflects the diligent effort of the Coomlttee since Its 
inception in March, 1970. 

The Committee is composed of three members representing business manage- 
ment, Mr. Morris Currey (resigned May, 1971), Mr. Allen Glusburgh and 
Mr. Leo Newsome (resigned November, 1970); three members representing 
economics and finance, Dr. Lewis Kohler, Mr. Clarence Nevrby, C.P«A« and 
Mr. Alvln Rosenthal, C.P.A.; and three aembers from the learning sciences. 
Dr. John Geddes, Mr. Donald Kitch and Dr. Agnes Robinson. As chairman 
I am Indebted to them for their time and conscientious labor. 

The Committee wishes to thank the State Depurtment of Education for its 
assistance and cooperation in this endeavor. 

Respectfully submijtted. 




Alvfn L. Rosenthal , /Chairman 
Advisory Connittee on Program 
and Cost Effectiveness 
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REPORT TO THE STATE BOARD PROM THE ADVISORY COMMITTEE 
ON PROGRAM AND COST EFFECTIVENESS 



The enabling legislation, AB 606, (California Legislature 
1969) as modified by AB 1923 (California Legislature, 1970) 
states: 

"The Advisory Committee on Program and Cost Effec- 
tiveness shall develop and recommend to the State 
Board of Education a methodology for evaluating 
^ the cost effectiveness of projects financed by 

^ Titles I and III of the Elementary and Secondary 

^ Eduoation Act of 1965, Miller-Unruh Basic Reading 

^ Act of 1965 

^ DEFINITION OF COST EFFECTIVENESS 

^ In the context of this report cost effectiveness may be 

defined as a management decision process which allows the deci- 
sion maker to choose from among feasible alternatives on the 
basis of least costs and maximum program output as measured by 
student achievement. 

DELIMITATION OF COMMITTEE'S TASK 

This report will summarize the findings of the cCinmittee 
for the first calendar year of the operation. Pursuant to the 
legislative mandate, this report will be limited to findings 
and recommendations necessary to develop cost-effectiveness 
methodologies for educational programs financed under ESEA 
Title I, ESEA Title III, and the Miller-Unruh Basic Reading Act 
of 1965. Further, the content of this report is focused on the 
reading and mathematics components of the above-mentioned educa 
tional programs and the pupil populations served by these 
progreuns . 
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To the best belief of the conmittee, the findings and rec- 
ommendations in this report are congruent with the findings of 
other statutory committees and commissions whose functions are 
related to this committee and with the plans of the California 
State Department of Education. The findings and recommendations 
contained in this report are based on the testimony, exhibits, 
and investigation of the committee and are presented for the 
guidance of the State Board of Education. 

DEl'ELOPMENT OP COST-EFFECTIVENESS MODEL 

The development of the cost- effectiveness model is contin- 
gent upon these five components which, taken together, will 
produce a common methodology for the estimation and evaluation 
c;^ the relative cost effectiveness of educational programs in 
reading and mathematics. These are as follo\vs: 

1. Establishment of terminal objectives by grade 

2. Measurement of pupil achievement evaluated for 
content validity 

3. Weight of pupil characteristics - establish research 
on which characteristics have effect on learning 

4. Resource allocation - (direct and indirect dollar cost) 

5. Documentation - all proc^ram elements, methodology, 
staffing, etc. 

It ie the aonaluHor of the aommittee that without the develop- 
ment of theee conetituent parte a model aa euah cannot be 
developp.d. Consequently, the committee has focused on the 
description and explication of these component parts as a neces- 
sary first step to the development of a cost-effectiveness meth- 
odology. This report presents these findings and recommenda- 
tions . 
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FINDINGS AND RECOMMENDATIONS 



Finding No . ^1 

While individual teachers, schools and districts have 
developed specific performance objectives against which to mea- 
sure the achievement of their students in reading and mathemat- 
ics, there is not within the state of California a common set of 
achievement objectives in reading and mathematics for grades 1-8 
by which the output of these programs can be compared. 

Reaommendation for Finding No. 1 

There -should he eetabliehed for each aomponent of the read- 
ing program (e.g,, word study ekillBt oomprehcneiont and refer- 
ence akilla) and for eaoh component of the mathematioe program 
(e.g.t sets, numbers and numerals and geometry ) , statements of 
t'^.rminal objeativee by gradet for grades l-B. These objectives 
must he stated in measurable terms and should be derived from 
the basic state-adopted textbooks for reading and mathematics. 
The implementation of this recommendation in no way limits indi- 
vidual schools and districts in the selection of materials, 
methodologies t and development of performance objectives for 
students essential to meet th3 pupil and teacher needs repre- 
sented by individual school distriai^T " 



Finding No. 2 

Recognizing the existence of the Miller-Unruh state achieve- 
ment tesfting program for reading in grades 1-3 and the mandated 
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state achievement testing program for reading and mathematics in 
grades 6 and 12, there still is little consistent or systematic 
procedures for the measurement of the change in pupil perfor- 
mance that can be directly related to the instructional program 
of individual schools. Further, there is little relationship 
between existing measurement practices in schools, the reading 
and mathematics curriculum embodied in the basic state-adopted 
textbook and the terminal objectives indicated in Finding No. 1 
above. 

Recommendation for Finding No. 2 

There should be eatabliehed in the state of California an 
improved method for aystematioatly evaluating student achieve- 
ment in reading and mathematics through the use of standardized 
testing instruments which have the highest possible relationship 
to the terminal objectives for reading and mathematics referred 
to above. In addition, classroom teachers should be encouraged 
to evaluate their programs during their instructional process 
with locally constructed tests (criterion referenced tests). 
Periodic reviews of the standardized tests should be made by the 
State Department of Education to insure a continuing high rela- 
tionship to the terminal objectives of reading and mathematics 
referred to in Finding No, 2 above, and f.hanges in basic state 
textbooks for reading and mathematics a'^ adopted by the State 
Board of Education, 



^ COM WMl*»* 

Finding No. 3 

Individuals as well as pupil population differ in their 
ability to profit from instructional programs. Some of these 
differences ^variables) are: (1) socio-economic background; 
(2) ability to communicate in English; (3) educational level of 
the parents; and (4) family mobility. In evaluating the programs 
in reading and mathematics it is necessary to identify those 
differences (variables) which are significant in the pupil's 
attainment of terminal objectives in reading and mathematics. 
Determining the relative significance of these differences 
(variables) requires the development of a method for weighting 
them in terms of their effect on pupil achievement in reading 
and mathematics. 

Reoommendation for Finding No, S 

A thorough study of available reeearoh reportB ie needed to 
identify the most important of theee variables and to provide the 
baot'e for weighting them in terms of their relative eignifioanoe , 
Thie will make it poeoible to incorporate them into a ooet-effeo" 
tiveneae methodology along with the reeulta of the evaluation 
dieoueeed in Finding and Reoommendation No, 2, 
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Finding No. 4 

From the evidence heard by the committee^ it is clear that 
there does not exist at this time an adequate method by which 
direct and indirect costs (resource allooatlons) for a partiou-* 
lar instructional program may be readily determined At the 
school and at the classroom level. 
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BEST COPY AVAILABLE 

Beaommendation for Finding No, 4 

In order to ^provide the data neoea(\ary in evaluating pro- 
gram and ooet effeotiveneee, a revision of the etate aooounting 
manual and guidelines for Title I, III, and MiXUr-Unruh programs 
ehould be made, utilizing the experienoe gained in oommeroial and 
industrial cost aooounting applications. 

The forthcoming Planning, Programming, Budgeting System 
(PPBS) accounting system must be supplemented by school dis- 
trict's accounting departments to the end of developing more 
detailed direct and indirect costs extending down to the class- 
room level. The State should provide leadership i^n develr^ping 
and promoting the use of such accounting systems. 

In addition, it is recommended that programs funded with 
categorical aid should require as a condition to their funding 
provision for the establishment of specialined cost aooounting 
systems and the reimbursement for same, in order to provide the 
data needed to develop cost factors at the school and classroom 
levels. 

Finding No. 5 

There is a critical need to reproduce co8t->e£fective proarams 
to prevent unnecessary duplication of effort, and to disseminate 
these findings. For this to be accomplished, full documentation 
of program components, evaluation techniques, and identification 
of direct and indirect costs (resource allocations) is necessary. 



hh- 

'88 



H eaommendation for Finding No, 5 ^^^^ ^^^^ AVAILABLE 

The State Department of Eduoation must require that each 
program applicant submit an application and annual report whiah 
includes the description of all program and cost factors in 
ouffioient detail-, In addition j the State Department of Educa- 
tion should develop procedures for the dissemination of informa- 
tion on programs which have been determined to be cost effective. 



CONCLUSIONS 

These findings and recommendations synthesize the reports 
prepared by the subcommittees on planning, objectives, measure- 
ment, weighting, finance and accounting, and the meetings and 
work of the committee as a whole. The subcommittee reports and 
minutes of the committee are available. 

This report concludes Phase I of the charge given to the 
Committee on Program and Cost Effectiveness by the California 
State Legislature and embodied in the Education Code, 
Section 6499.204. 

With acceptance of this report, the committee would like to 
institute work on Phase II, the production of cost-effectiveness 
models based on data ob*- ined from experimental and pilot pro- 
grams, funded from the various sources of categorical aid, 
including State funds. 

Alvin L. Rosenthal, Chairman 

John A. Geddes, Vice-Chairman 

Allen S. Ginsburgh 

Donald E. Kitch 

Lewis T. Kohler 

Clarence Newby 

Agnes S. Robinson 
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Rationale for Project 

Phase II of the Work Plan called for a study of the several com- 
ponents that the Committee felt to oe important in the development 
of the methodology for assessing cost effectiveness. The Committee 
had spent a considerable amount of time during Phase I becoming 
acquainted with the scope of the problem and had developed some 
hypotheses as to the kincs of factors to be considered in any 
methodology that was to be developed. 

In order to test these hypotheses and gather further data, the 
Committee felt it would be helpful to have access to a school 
district which could serve as a laboratory in which to collect, 
examine and analyze data. The Riverside Unified School District 
was asked to participate in this venture because it had success- 
fully completed a project in reading and math and this would allow 
for the problems of replication, as well as the documentation of 
the several components of the project, to be studied. 

Expectations from the Project 

It was recognized that the specific results obtained from the 
Riverside project could not be generalized to apply to all other 
situations or be the sole source of data for the Committee. It 
was also understood that the actual achievement scores obtained 
by the students were not as important as the procedures developed 
for the collection and processing of data from the project. The 
Committee did not expect a methodology for assessing cost effec- 
tiveness to emerge full blown from this project. What it did 
expect was that by the use of live data in a real school setting, 
the Committee might oJ:>tain a better understanding of the elements 
being studied and identify practical problems that would not 
otherwise be discovered. It was also hoped that the results of 
this project would give direction to the establishing of a base- 
line for future studies regarding the relative importance of 
selected variables in the pr^adiction of performance in reading 
and mathematics and to the development of a methodology for 
assessing program cost effectiveness. 

The specific outputs expected from the Riverside project included 
the following items: 

1. A cost system that would be capable of identifying 
program costs at the subject, course, component and 
subcomponent levels, and a description of how each 

of these cost units would be identified and collected. 

2. A listing of instructional objectives. 

3. An evaluation design. 
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The coiloction of data pertaining to studont 
charucLori:jLica including such items a.;: 

«». I.oi (|f -'rhoi 11(1 i k«' IQ acoroa; vitIi.iI, 
nonvi • rbul , toLu I . 

b. Socioeconomic index based on father's 
occupation. 

c. Mobility index. 

d. Sex. 

e. Chronological age. 

f. Number of siblings in the home. 

g. Number of unexcused absences. 

h. Ethnic background. 

i. Other data as deemed appropriate. 

The collection of data pertaining to teacher characteristics 
including such items as : 

a. Age. 

b. Sex. 

c. Years of teaching experience. 

d. Number of units of professional courses. 

e. Number of units of academic courses. 

f. Salary level. 

g. Number of years in the district. 

h. Number of units tadcen in college after employment 
in the district. 

i. Selected dimensions of teaching performance as 
deemed appropriate. 

Documentation of all aspects of the project necessary for 
understanding and replication including: 

a. Preliminary planning j.-equired. 

b. Staffing. 

c. Course content. 




d* Teaching strategies. 

e. Classroom organization. 

f. Evaluation criteria and measuring devices. 

g. Staff inservice training required. 

h. Procedures for collection of all data. 

i. Treatment of data. 

j . Other items as required for project 
replication. 

7. Analysis and statistical treatment of the data collected 
with interpretation as appropriate. 

Project Design 

Following the selection of Central Junior High School as the project 
site and the approval of funds for the project, Riverside Unified 
School District submitted a research desian which was accepted by 
the Committee. 
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RIVERSIDE UNIFIED SCHOOL ^.ISTRICT 
Riverside, California 



October, 1971 



RESEARCH DESIGN FOR THE 
COST EFFECTIVENESS STUDY 



Introduction 

Enabling legislation, A.B. 606 (California Legislature, 1969) 
modified by A.B. 1923 (California Legislature, 1970) directed 
the Advisory Committee on Program and Cost Effectiveness to 
"develop and recommend to the State Board of Education a method- 
ology for evaluating the cost effectiveness of projects financed 
by Title I and III of the Elementary and Secondary Education Act 
of 1965, Miller-Unruh Basic Reading Act of 1965 ..." 

Phase I activities of the Commission resulted in the description 
and explication of five components necessary to the development 
of a common methodology for the estimation and evaluation of the 
relative cost effectiveness of educational programs in reading 
and mathematics. 

Phase II activities of the Commission will include the production 
of cost effectiveness models based on data obtained from experi- 
mental and pilot programs. 

With the cooperation and support of the California State Depart- 
ment of Education, a Demonstration Program in Intensive Instruc- 
tion in Reading and Mathematics for Low Achieving Pupils, as 
authorized under Education Code 6490ff., was approved for the 
Riverside Unified School District. Funds amounting to $84,469 
were allocated for the study . Central Junior High School in 
Riverside was selected as tha site for the projoct with the total 
seventh grade class (approximately 370 sf-^'dents) forming the student 
population. Eleven teachers and the chairmen of the English and 
mathemu*:! cs departments will be included. 

An effort will be made to ascericain the relative importance of each 
of several student related variables, teacher characteristics, and 
components of instruction to the prediction of performance in 
reading and in mathematics within the framework of cost effectiveness. 



Objectives 



' The overall objectives of the study are: 



1. To develop a model system for determining the cost 
effectiveness of public school instruction at the 
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classroom ievtji. Tne system will include 

a. tne specif icatior\ of instructional objectives, 
D. tho devt.lopinent and application of an evaluation 
uosiqa, 

c. tne collection of pertinent student and teacher 
characteristics , 

d. the development of a cost accounting system 
capable of generating all applicable direct 
and indirect costs, and 

e. procedures for documenting all components. 

2. To assess an ongoing seventh grade program of reading and 
mathematics in order to demonstrate the applicability of 
the model . 

3. To determine which variables of students and teachers, and 
of the instructional programs involved are the most signif 
icant for success. 



Analysis of Data 



A multiple regression modsl involving double cross validation will 
be employed relative to each criterion vari2ible and selected combi- 
nations of predictor variables— -the selections being based on both 
empirical information derived from the intercorrelatior.i of the pre- 
dictor variables and from their correlations with the criterion 
measures and logical analyses of the cost factors involved in the 
obtaining of information on the predictor variables. The double 
cross validation procedure will necessitate taking two samples con- 
sisting of odd numbered students and even numbered students, of 
working out for each sample the same sets of multiple regression 
equations, of comparing the relative contributions of each of the 
independent variables to the criterion variables in each set of 
equations, and of employing the regression equation derived from 
one group upon the other group to ascertain how much shrinkage 
because of sampling factors occurs in the multiple correlations 
(of predicted scores for one group from the equation of the second 
group with the obtained scores of the first group) . Stepwise 
multiple regression procedures will be used which permit the 
selection of composites of variables in order of the contribution 
of each variable to the prediction of the criterion. Logical selec- 
tion of variables will also be made in terms of certain practical 
considerations . 



Limitat i ons of the Study 

The use of only one school within a given district leads to certain 
disadvantages or limitations in an accountablility study. There 
will be no basis for comparison of what happens in the school 




involved with other schools or districts. Consideration cannot be 
qiven to differences or variations in variables such as class size, 
tax rates, assessed valuations, ADA, rate of staff turnover, and 
salary scheaules. These variables are in essence fixed. The data 
obtained in the present study will probably use the individual 
student as the sampling unit, and any gains studied are likely to 
be highly unreliable in view of the ^act that they are primarily 
directed toward the individual student or a few classes. 

On the other hand, a relatively high degree of control can 
probably be realized in this investigation, as the project per- 
sonnel can monitor what goes on in the clcissroom. 



Vari ables in the Investigation 
Dependent Variables 

The dependent variables will consist of reading and mathematics 
portions of the Comprehensive Tests of Basic Skills, Level 3, 
Form Q. These tests are being administered on both a pre and posttest 
basis. In addition, several of the teacher-made tests which have 
been anchored to carefully prepared behavioral objectives, will 
also be used as criterion variables. Scores used will be obtained 
from tests administerea at the end of the school year. The pretests 
may be used also in the context of predictor variables. 



Independent Variables 

Tne independent variables will consist of those pertaining to stu 
dent characteristics, teacher characteristics, and components of 
instruction. Suggestions are presented below of several possible 
pi.dictor variables. 

-'^ Student characteristic s — Among the student- related variables 
will be: 

U) Lorge-Thorndike IQ scores — Verbal, Nonverbal, and/or 
Total ; 

(2) socioeconomic status based on father's occupation; 

(3) mobility index; 

( 4 ) sex ; 

(5) chronological age; 

(6) number of siblings in home; 

(7) number of unexcused absences; 

(8) intellectual achievement responsibility as measured 
by The Intellectual Achievement Responsibility 
Questionnuire (Crandall, Katkovsky, and Crandall) ; 

(9) need achievement as measured by Mathematics Self- 
concept Questionnaire (Holly) ; 

(10) study attitudes and methods determined by modifica- 
tions of Stuciy Attitudes and Methods Survey (Michael, 
Michael, and Zimmerman). 
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Teaqher charactor;i_s_tics--The following teacher attributes 
will be considered "as variables to serve as predictors: 

(1) age; 

(2) sex; 

(3) years of teaching experience; 

(4) number of units of professional courses; 

(5) number of units of academic courses; 

(6) salary level; 

(7) number of years in district; 

(8) number of units taken in college after employment 
in district; 

(9) selected dimensions of teaching performance as 
determined by student evaluations. 

The inclusion of this information iii the regression equation wiJl 
be complicated since each student who has a given teachor will be 
assigned the same score on the relevant variable for that teacher; 
hence the variance may be limited, or separate analyses may have to 
be made for groups of students who have teachers with common charac- 
teristics. Possibly two groups of teachers who represent marked 
contrasts in certain combinations of attributes will be associated 
with subgroups of students for whom separate multiple regression 
analyses will be made. Discriminant ciaalyses may also be employed. 

Components of instr u ction — Variables within classrooms 
include numbers of students eind amounts of instructional 
time spent in 

a. large group instruction 

b. small group instruction 

c. laboratory experience 

d. independent work. 

In addition, each classroom will be rated in the following 
four principal areas of educational procedure 

a. individualization 

b. interpersonal regard 

c. creative expression and divergency of thinking 

d. group activity. 

Anc'lysis of the data described above will result in a detailed 
assessment of the cost effectiveness of the Central Junior High 
School seventh grade instructional programs in two basic subject 
areas. From the results of the study it will be possille to suggest 
a model system to be used elsewhere for a similar purpose. 

Inptructional Program 

The content areas to be covered in the project were reading and 
mathematics, and the student population was the total seventh grade 
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at Central Junior High School. The distinctive aspect of the pro- 
gram was the manner in which individualization of instruction was 
accomplished. 

In tne Reading program the curriculum was carefully analyzed and 
specific performance objectives were written for each of the areas 
of skills or knowledge where it was desired that student learning 
take place. Criterion-referenced tests were developed to assess 
tne degree of achievement for each of the objectives. Teaching 
strategies included the use of Learning Activity Packages (LAPS) 
designed to teach each of the objectives, large group instruction, 
small group instruction, independent study and laboratory work 
with such machines as the Hoffman, controlled reader, cassette 
players, filmstrips and reading pacers. 

Vhe Math program attempted to individualize the program by grouping 
wxthin the classroom and careful record keeping of individual stu- 
dent progress as a basis for assigning and reassigning tc groups. 
Lxtensive use was made of calculating machines. 

Examples of performance objectives and criterion-referenced tests 
are included here, and complete sets are on file at the Riverside 
School District and in the Committee's permanent file. 
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Directions: Read the story. Decide which of the words given 

below belongs in each sentence. Pill in the space 
by the correct letter for the answer to each ques- 
tion. Use answer sheet B. 



WORKER DIAMONDS 



Glittering gems called diamonds are among the most 
precious of human possessions. Through the ages 
they have been a lasting evidence of wealth. Lands 
may lose their soil, buildings may be destroyed, 
stocks and bonds may become worthless, but the value 
ot diamonds remains relatively unchanged. 

Diamonds are not of value only as signs of wealth. 
In addition, the diamond is one of the hardest, 
longest-wearing substances known to man. Because 
^Ll L tt ""^"^ valuable for practical use. In 
tact, about three- fourths of the total annual supply 
of diamonds is used in factories and machine shops. 

A diamond on the end of a cutting tool can cut through 
the haraest steel, but only a diamond can cut another 
diamond. Because of their unusual hardness, diamonds 
*^\k^® -^^ sharpen grinding wheels. They are placed 
on the tips of the grinding drills used to cut through 
tons of bed rock, in scores of other ways, diamonds 
are essential to turning the wheels of modern machinery. 

About 4 1/2 tons of diamonds are mined annually. The 
vast diamond mines of South Africa produce most of 
these diamonds, but in recent years many diamonds 
nave also come from Brazil. 

Questions: 

^' known^tS°Ilan!^ °^ ' longest-wearing substances 

2. In recent years many diamonds have come from 

3. Only a can cut another diamond. 

^' i^°used^ir~^°''''^^^ ^'^''''^^ ^""PP^^ °^ diamonds 
5. The value of diamonds remains relatively . 



^* _j — called diamonds are among the most precious of 
human possessions. 



ERIC 



70- 



7. A diamond can cut through the hardest 

8. Diamonds are used to grinding wheels. 



9. About 4 1/2 tons of diamonds are 



annually . 



10. Diamonds are 
machinery . 



to turning the wheels of modern 



Choose from these answers. 

A. factories 

B. sharpen 

C . gems 

D. hardest 

E. mined 



Mark answer sheet. 

F. Brazil 

G . unchanged 

H. steel 

I. essential 
J. diamond 



Test Filing No. LC lp/1/10-10 
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Behavioral Objectives for Lovel A of the 
Prescriptive Mathematics Inventory (PMI) 



NUMERATION SYSTKMS 



^orm Item No. Pictorial Sets 



1. Given a set of less than. 10 elements, the 
student will count the elements, 

2, Given a set containing whole numbers as ele- 
ments, the student will identify all elements 
that are "more" or "less" than a given number. 



Place Value 



3 3 3, Given a one-, two-, or three-digit number, tiie 

student will specify the value of each digit in 
the numeral. 

^ 4 4. The student will specify tne value of each 

digit in a four- or five-place numeral. 

3 5 5. The student will specify the value of each 

digit in a numeral of nine or fewer digits. 

3 6 6. The student will specify the value of each 

digit in a decimal number between 0 and 0.999. 



3 7 



The student will specify the value of each 
digit in a decimal number of four or five 
decimal places. 



Expanded Notation 

8 8. The student will formulate a three-digit 

numeral from the expanded form of the number. 

9 9. The student will formulate a four-digit 

numeral from the expanded form of the number, 

10 10. The student will formulate a five-digit 

numeral from the expanded form of the number, 
which form uses exponents to the base 10. 



A-1 
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Costs 



The development of the accounting system for this project was a 
joint effort on the part of Mr. Walter Parks, Business Manager of 
the Riverside Unified School District, and his staff, together 
with Mr. Clarence Newby and members of his Sub-committee on Costs 
of the Advisory Committee on Program and Cost Effectiveness. The 
cost system was developed for the purpose of identifying and 
collecting costs in sufficient detail so that units within the 
classroom could be identified and costs determined. A format of 
reporting these costs, if used by other projects, would allow for 
a cost comparison of total programs or equivalent parts of 
different programs. The system was designed to be compatible 
with the PPBS, being designed at the State level. The cost 
system, as developed for the Riverside project, has been reported 
in a previous chapter under the report of the Sub-committee on 
Costs. 
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COST SYSTEM OBJECTIVE 



To develop a cost system capable of generating all 
applicable direct and indirect costs for a school 
district to the following levels: 



Department or Project 
Location 
Function 
Life Span 
Program 
Course 

Course Components 
Sub Components 
Objectives 
Sub Objects 
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COST SYSTEM OUTLINE 



1. Statement of objectives 

2. Procedure statement 

3. Final report format 

4. Explanation of function headings 

5. Account number structure 

6. Account number example 

7. Time card Scunple 
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COST SYSTEM 



General Procedure 



o^L^tf-hr^ designed for the AB 938 Cost Effectiveness pro ject 
ll^rll ^ development of both direct and indirect costs fLt. all 
sources for any desired level of the instructional program! The 
st^ScSrf centered on the course level of the total program 
structure and the final reporting will be on a per course basis. 

Levef V^E^ tJjo'.Jih • ""^^ ^^^^ ^^^^^^ structure, the course level is 
ture rlkdinn ^^^^^ ^.^^ sub-classif ication. Journalism, litera- 
ture, reading and speech would be Level VI courses within the 
English program or department, Lev>l V. witnin tne 

aeJerv??f^or ^5.?°?'^^^^ breaking each course into components 
jLevel VII) or even into sub-components (Level VIII) if necessarv 

desirab?rto"mai^^' subdivision of a course for .Lc" rtl7' 

t^?on^°anTt^it;r1a1^?L\^?=n?" ^^^^^^^^ .To^lnltl^l 

search anS^S2r^^ ''^'^^ ^''^"^^ strictly instructional such as re- 
R^^f^^^h included in the program structure. 

exoeriSen?2? frni?^'"^"^ ^'"P^^^^ ^h^" the project is 

mod?ft«r u tu f °^ ''^^^ existing course is imposed or 
SSiia" r^s Jrn^ ""^^ ^°-called "tool up" and the "on- 

^ninono^rr ^w^""^ development costs, are combined into one 
component for the AB 938 Cost Effectiveness program currently 
under study at Central Junior High. Program currently 

twn iinl\ ^o^^J^f ^"^"^ Central's program would consist of 
=nSmfi??L^2 reading and math. Costs are 

n?J^ finctinS^?^^^^^ component on the final report form by 

(2ri^s2rSc??oL?^™fi"^'-, ''^??f instructional salaries, 

testing materials, (4) project 

(8r 2i^looi?atfn^^^^ 5°°^^' equipment, '( 7) facilUies, 

(8) transportation, and (9) administration and services. 

severa! iLms^hf.h^" °f function headings, which include 
acSaipd in """"^^ prorated, all of the costs cannot be 

inHrocess T^er ^^^^^^ "^^^^^ account- 

Therefore, a separate cost ledger must be maintained 
and entries made from several sources for each course. The finSl 
report is prepared from the summarized cost-ledger data. 

Also proration schedules must be established and maintained for 
:r?S r?rfe-areaiL\'H' P'""'' equipment, books, and\ela?ed'?tems 
life Llhlr ??^n ? ^-^^ ^"^^^ prorated over their useful 

life rather than lumping the entire cost into the year of purchase. 
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When a course or project is first brought into the cost system, 
an inventory of equipment and other assets of significant value 
must be inventoried, valued, and a proration schedule established. 
The^e on-hand assets then can be regularly charged to the cost 
ledger via the proration schedules. 

This system requires that time sheets and time and material sheets 
be maintained to develop the cost of instructional salaries for 
each component of a course and other direct costs from sources 
within the district. 

Entries into the cost ledger then will be from several sources in- 
cluding the appropriation ledger, the proration schedules, the 
time sheets, and the time and material sheets. 
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EXPLANATION OF FUNCTION HEADINGS 



INSTRUCTIONAL SALARIES 
Definition 

Instructional salaries are defined as the salaries of those indi- 
viduals who are directly involved with the teaching of students. 
This would include the salaries of teachers, classroom aides, spe- 
cialists working within the classroom, and counselors if working 
on course-related problems. Teacher and aide salaries are charged 
at a standard cost rate. Other salaries are charged at the actual 
rate of pay for each individual. 



Source o f Cost Information 

Instructional salaries for each course and course component are pre- 
determined as closely as possible as part of the budgeting process. 
The final budget will reflect in detail these predetermined estimates. 
During the year each individual involved will submit time cards 
Weekly detailing the actual time involved with each course or com- 
ponent. See sample time card. These are posted directly to the cost 
ledger. 



Applicable Account Nur ^ bers 

Teacher 
Aide 

Counselor 



0213.094 XX XXX xxx 
0220,194 XX xxx xxx 
0214,294 XX xxx xxx 



li^STRUCTIONAL MATERIALS 
Definition 

Instructional materials are defined as those items purchased spe- 
cifically for a course which would ordinarily be consumed within one 
year or for which the primary period of use is during the first year 
after purchase. These include materials, periodicals, magazines, 
workbooks, postage, film rental, travel conferences, hand tools and 
low-cost equipment. Large expensive equipment are excluded from 
this category. 



Source of Cost Information 

Most instructional material costs are charged directly to the appro- 
priation ledger from the purchase orders. They are accumulated 
automatically by account number as part of the bookkeeping process. 
Charges which have to be prorated among several accounts can be 
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accomplished with journal entries to the appropriation ledger or 
with prorated entries to the cost ledger. 



Applicable Account Numbers 

Instructional Supplies 

Mileage 

Conference 

Printing 

Magazines 

Workbooks 



0290.194 XX XXX xxx 
0291.194 XX xxx xxx 
0291.294 XX xxx xxx 
0291.394 XX xxx xxx 
0292.594 XX xxx xxx 
0292.694 XX XXX xxx 



TESTING MATERIALS AND SCORING 
De finition 

3h?^h"^ materials are those pre and post tests plus related materials 
coirse ^^""^ °^ measuring process for an individual 

Scoring includes the costs of service agencies, district computer and 
computer personnel time, and the costs of hand scoring. 

Source of Cost Information 

The costs of tests, materials, and service agencies are charged from 
the purchase orders directly to and accumulated in the appropriation 
leager, the same as Instructional Materials described above. 

District empioyees involved with computer or hand scoring must sub- 
mit time and material sheets detailing the time involved with each 
course or component. These are posted directly to the cost ledger. 



Applicable Account Numbers 

Testing Materials 
Scoring 



0294.194 XX xxx xxx 
0294.294 XX xxx xxx 



PROJECT ADMINISTRATION SALARIES 
Definition 

n.!:^Vf^® group of courses may be designated as a special project 

usually with special funding. The project director is defined as 

^nfjvidual directly responsible for all activities of the proj- 
ect, other examples of project administration salaries include 
specialists, secretaries, evaluators, and cost accountants. 



Source of Cost Information 

Project administration salaries for each course and course component 
are predetermined as closely as possible as part of the budgeting 
process. The final budget will reflect in detail these predetermined 
estimates. During the year each individual involved will submit time 
cards weekly detailing the actual time involved with each course or 
component. These are posted directly to the cost ledger. 



Appli cable Account Number 

Project Administrator 0212.043 xx xxx xxx 
Secretary 0220.043 xx xxx xxx 



BOOKS 
Definition 



Books are defined as all textbooks, supplementary books, and library 
books with a useful life of greater than one year. Pocket books, 
workbooks, and other book-like materials are charged to Instructional 
Materials if their useful life is less than one year. 



Source of Cost Information 

The total cost of the books will be charged to the appropriation 
ledger from the purchase orders. The total accumulated cost for 
each category is divided by the average life for textbooks. The re- 
sultant amounts are set up in a proration schedule and charged accord- 
ingly to the cost ledger. 



AppliccUole Account Numbers 

Textbooks 0230.194 xx xxx xxx 

Supplementary books 0241.194 xx xxx xx:: 

Library books 0240.194 xx xxx xxx 



EQUIPMENT 
Definition 

Equipment is defined as physical property with a relatively high per- 
manent value. Equipment is usually movable in contrast to building 
fixtures or site improvements. Items of relatively low cost which 
are consumed for the most part within one year should be charged to 
Instructional Materials. Repairs to equipment are also part of this 
function. 
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Source of Cost Information 



The total cost of any equipment and replacement equipment will be 
charged directly to the appropriation ledger from the purchase 
orders. Charges are made each year from the proration schedule to 
the cost ledger. For each piece or group of equipment, a proration 
schedule must be established, based on the estimated useful life of 
the equipment and its total cost from the appropriation ledger 
less its residual value. 

Equipment on hand at the beginning of a course or project should be 
inventoried and a proration schedule developed based on the re- 
maining value and useful life. Equipment transferred into the 
course during the year from other sources should be similarly valued 
and charged. 

Repairs by outside vendors are accumulated directly in the appro- 
priation ledger from purchase orders. Work performed by district 
personnel must be recorded on time and materials sheets from which 
the costs are transferred to the cost ledger. 



TRANSPORTATION 
Definition 

Transportation is defined as the cost of transporting students for 
.reasons specifically related to the individual course. This in- 
cludes primarily field trips and transportation between schools for 
the purpose of attendance in certain courses. Ordinary home-to- 
school transportation is excluded. This cost is part of the 
prorated administrative and service costs. 



Source of Cost Information 

Field trips and other special trcunisportation are charged directly 
to the appropriation ledger from field trip purchase orders. These 
are accumulated automatically as part of the bookkeeping process. 
Charges which have to be prorated among several accounts can be 
accomplished with journal entries to the appropriation ledger or 
with prorated entries to the cost ledger. 



Applicable Account Numbers 

Field trips 0590.294 



Applicable Account Numbers 
Equipment 

Repairs to instructional equipment 



1272.094 XX XXX xxx 
0791. Ixx XX xxx xxx 



ERIC 




FACILITIES 



Definition 

Facilities are defined as all of those costs related to providing 
space for a course or project. These include the building, the 
building fixtures, the maintenance and custodial functions, and 
utilities. Ordinary classroom furniture may be considered a 
facility cost rather than an equipment cost. Furniture purchased 
for or peculiar to a specific course or project would be charged 
under equipment. 



Source of Cost Information 

Building and building fixture costs are developed from a standard 
districtwid* per square-foot charge. A computation will be made 
for each coi rse and component and entered directly into the cost 
ledger. 

Maintenance and custodial costs are based on a districtwide per- 
pupil cost for the Maintenance and Operations department. The 
per-pupil cost times the number of pupils involved in each course or 
component are entered directly into the cost ledger. 

Utilities costs are accumulated in the appropriation ledger by 
school location. These are prorated to each course on a per-pupil 
basis. 



ADMINISTRATION AND SERVICES 
Definition 

Administration and service costs are defined as all other costs not 
covered in the previous categories. These usually are noninstruc- 
tional school and overall district costs which are prorated to the 
individual course or component. 

Included are principal, vice principal, secretary, superintendent's 
office, business office, psychological services, attendance, audio 
visual, personnel, home-to-school transportation, and other admini- 
strative and service departments. 



Source of Cost Information 

All administrative and service costs are prorated on the basis of 
cost per ADA. School administration costs would be divided by the 
total school ADA and the resultant per ADA cost is charged to the 
individual course or component. In this same manner, district costs 
are divided by the total district ADA and the resultant per ADA cost 
charged accordingly. 



ERIC 



-82- 
111 



0) c 



nj 



0) 

if) 



4J 
a 

0) 

u 

0 

CO 



8 



4J 

Q 



0) 

(J) 
u 

8 



I 

u 
o 

c 
o 

o 



4J 4J 

C3 O 

e •rn 
•P O 

Q 0 



4J 
O 
0) 

cn O 



0 



tn 

0) 

> 

O r 

o < 



I 

cu 
o 

H 

0) 4J 

0) § 
Q g 



•H ^ 

'd o 

Pi4 



•H 
H 

M 



a 

0) 

o 



00 



(A 
0) 

a 

O C4 

(0 



to 

0) 
•H 

H O 

B' 

CO 



o 

C 

O ON 

o 



0) 
M 

U 

(U 



0) cu 
u o 
hi) o 



o 



J1 

•H 



M H 
4J 00 



5 



o 

•H 

a 

4J OS 
(Q 



2 

0) 
0) 



o 



•0 

0 
•H 



(d r 
H 



I 

00 

I 



§ 

•H 

•P 
O 



4J 

(0 

c 

H 



o 
o 



r4 

4J O 

H 



ERLC 



lis 



LAB REPORT SHEET 



Student Teacher 

Period 



Each time you work in the lab, please specify the date and check the type 
of materials with which you worked. 



Date 


Machines 


P iroar dinnied 
Materials 
(LAPS, etc.) 


Worked with 
Tutor 


Dth^r—Soeci f V 


Hoffman 


Controlled 
Reader 


Cassette 
Player 


Fi Imstrip 


Pacers 
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RECORD OF PERFORMANCE ON CRITERION-REFERENCED TESTS 



ocuaonL __ Teacher 

Period 



DiiJiiCTIVii 


CTBS 


Date 


P/PP 


Test 
No. 


Pass/ 
Fail 


Date 


. ... 1 
P/PP 


Test 


Pass/| 
J'ail iDate 


P/PP 


•k w w 1^ 

No. 


XT do O / 

Fail . 


Literal 
"omorehension 
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RIVERSI DE UNIFIED SCHOOL DISTRICT 



Daites^ of Report 



COST EFFECTIVENESS PROJECT 
READ ING AND MATHEMATICS SEVENTH-GRADE-CENTRAL 
( Report in minutes) N ame of Teacher 



COMPONENTS 

T M r* T A C c ' 


IMon. 






Thu. 


IFri . 


REMARKS 


iiAKGt« GROUP 


! 


1 










oMALL GROUP 














A. 

iWDiVIDUAL INSTR, 














Li Ad work 














liARUti GROUP 














oMAjLili GROUP 














INDIVIDUAL INSTR, 














LAB WORK 












— — 


LARGE GROUP 


1 












• SMALL GROUP 














c 

* INDIVIDUAL TNQTP 














LAB WORK 














LARGE GROUP 














SMALL GROUP 














D. 

INDIVIDUAL INSTR. 














LAB WORK 














OUT OF CLASS 
GRADING 














RECORDKEEPING 














PREPARATION 














RESEARCH & DEVELOPING 















2'i ?i^qprffon°?'^ ^^^S'^f^cS? ^^^^^ " ^""^ ^"^^ Dawson's mailbox on Friday. 

2. ) In Section 1 - "IN CLASS" record time IN CLASS room as devoted to these 

components. Record each class period separately in groupingrA-B-C or D 

3. ) In Section 2 - "OUT OF CLASS" record time out of classrooms as devoted to 

^l?irar^??gS'^IS^^ilaL^?g^^^^j°U!"^^ ^^^^ that°ex^eS5eTirsh'???^?fn^^ 
^^'^ iSvf^Jt'i^Json^^fJJ*^ that time expended in planning, developing, research, 
ERIC the pJo|?^°Si coSrsef^ ""^^ P^^"^ for^mgiovement oi 
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METHODOLOGY 



The heart of the methodology for assessing cost 
effectiveness is the model used to conceptualize 
the problem and to specify and define the factors 
involved. The Committee heard a number of reports 
in the area of models and modeling and reviewed a 
number of specific models. An introduction to 
modeling and two specific models will be presented 
in this report that represent the direction of 
the thinking taken by the Committee, and a summary 
type report will outline the requirements of a 
model and the steps required to complete the model 
suggested by Mr. Ginsburgh. Other reports and 
models may be found in the permanent files of the 
Committee. 

The introduction to modeling was presented by 
Dr. William McCormick at the meeting of the 
Committee on December 21, 1970. It was in the 
form of an assessment of the current status of 
the Committee in relation to the work plan and 
directions in which to move in the future. 
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BEST COPK AWllABli 



Re vis ion of the iMas ter Plan 

Dr. McCorinick stated chat th3 Committee's Master Plan is a well 
tuouynt-out plan wuich has served us well and will continue to 
do so. lie recommended t\\ro revisions at. this time: (1) the two 
sub-committees presenLiy active (Scaling and Measurement) should 
try to advance the dato, by one month, for their concluding re- 
ports (to January and x'ebruc.ry) ; (2) the Model Planning Sub- 
committee should be app^ inted and start their work now. 

Two benefits are expected: (i) tne sub-committees would be 
three months behind the master plan, rather than four; and 
(2) the needs of the Model Planning Sub-committee could help 
focus the efforts of tne other two sub-committees. Rather than 
being three independent groups, the task of each depends upon 
the direction taken by the other. In particular, different 
kinds of models (and their purposes) may require different sets 
of dependent and independent variables (the concern of the 
Measurement and Scaling Sub-corairdttees) . 

The Process of Modeling the Compensatory Programs 



Using existing theory as a 
basis for program modeling, 
establish relationships and 
functions . t 



Gather data used as 
a part of the ad- 
ministration and 
evaluation of programs. 



Sources : 



(1) Literature 



(2) Evalu- 



ation and 

Program 

Consultants 




( Conceptual Models) 
Student Achievement = Function of 
(Yl,Y2,....i 



Data Base 



Select variables from the 
data base and conceive 
ot.hers as needed to define 
th9 conceptual models. 



( Operational Models) 



Test Achievement = X + X^^ + X2 + 



• • • 
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3 . Two General Approaches to Modeling 

There seem to Ji^e two general approaches for consideration. 
Should the efforts be (1^ conclusion oriented or (2) decision 
oriented? The former is more open and free to explore in- 
teresting relationships while the latter is more closed or 
limited to the requirements of decision makers. 

a. Conclusion oriented models are often used for investiga- 
tion of the relative effects of several forces which in 
unison determine the level of output. Such a model can 
also be used to predict an output (expected level of 
achievement) for a unit of analysis (say a school) . 

Such models can include many factors that are known (on 
basis of research) to contribute to student achievement; 
such as measures of the student's past learning rate and 
motivation, and the quality and intensity of his learn- 
ing program. These are factors that are said to predict 
his achievement . 

b, A decision oriented model may include a considerable 
number of factors but they are included for the purpose 
of increasing the precision of the conclusion; that is, 
the model's purpose is to indicate which practice is 
"best" given the other information in the model. 

These models can be limited to a very few (one even) 
factors and achievement can be contrasted for each 
alternative. Comparability is achieved by random 
selection of projects to study. The rationale is that 
all other causal factors are represented equally, and 
any test differences between each program alternative 
is due to program differences. 

For example, if the question was whether teacher aid 
programs were more cost effective than reading 
specialist ones, a random selection of each kind of 
project could be studied. Using equal cost levels, a 
random selection of each type of program could be con- 
trasted to see which had greater gains in student per- 
formance. A decision could then be reached, recommending 
the program with the greater gains per level of cost. 



4 
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4 . An Example of a Conclusion-Oriented Approach 



INPUTS 
Xi 



Xj^ student Types 



X2 Staff Types 



X3 Supply & Capitol 
Items 



X4 Instructional 
Approaches 



Model equation 
(Xq is the Y intercept) 



OPERATIONS 



Instruction 
"Black Box" 

Inputs are 
categorized 
and combined 
in various 
ways for 

different amounts 
of time 
according to 
planned 
sequences 



OUTPUTS 



Yi 



Student change 
in achievement 
in terms of 
what is 
"expected" 



Y=Xq+Xi+X2+X3+X4+E 



"Modeling" starts with stating a system of theoretical relationships. 
Measures must then be selected to describe the inputs, operations and 
outputs. The degree of controlability of operations must be specified; 
in addition, the decision-making process must be defined in terms of 
resource acquisition and allocation. 



Expected benefits : Answers to the question of how much "bang" for a 
buck (achievement gain for a dollar invested) for a particular school 
input, after controlling for the others. 

5. An Example of a Decision-Oriented Approach (Analysis of Variance) 





X2 Instructional Strategy 


Traditional 


Diagnostic- 
Prescriptive 


"Other"? 


LEVELS 
Xi OF 

COST 


Hi 








Medium 








Lo 









Model Equation Y=Xq + X^^ + X2 + E 



Achievement » The overall average + the level of cost + 
the instructional strategy + error (includes unknown 
factors) . 
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Units of analysis (children or schools) are 
randomly selected and sorted into the cells 
indicated above. Using the average achieve- 
ment and variance derived for each classifi- 
cation, the unique contribution of each factor 
(level of cost and instructional strategy) is 
calculated. 



Expected benefits ; An answer to the question of which instruc- 
tional strategy (of the two chosen) has the greater achievement 
associated with it, for a given level of cost. 

6 . The Mission and Constraints of the Committee 

The Committee is to recommend to the State Board of Education 
a methodology for evaluating the cost effectiveness of com- 
pensatory programs. It is important for us to consider its 
use; i.e., projects with low ratings are to be modified to 
produce a higher degree of program and cost effectiveness. 
Methodology that allows only a rating scheme will not provide 
inform ation for the purpose of modifying projects which are 
given low ratings. Yet, it seems that the Committee should 
be concerned with models (methodology) which will provide the 
required information. 

Of course other groups are seeking ways to determine the 
effectiveness of school programs (and their costs) . They 
include the State Department of Education, the Advisory 
Committee for PPBS, the State Committee on Goals and Evalua- 
tion, the Innovative Schools Act (AB 416), and the Legislative 
Analyst's Office. 

The Committee is restricted to convincing the Department of 
Education of the wisdom in expending its resources for cost 
effectiveness work (the 900 account) in the manner the 
Committee thinks best. 

7. Determining Project Effectiveness 

We can measure directly the performance of the student in 
reading and mathematics, but not that of the contribution of 
school efforts; that is, the unique effects of experienced 
staff and instructional techniques (such as phonetics) . 
Rather, these must be inferred from the empirical analysis 
of results obtained by different combinations of instruc- 
tional staff and techniques working in compareUale circumstances. 

Estimates of the effectiveness of educational projects must 
be relati/e to particular circumstances which individual 
schools, in the short run, are unable to change (the learning 
rates and style of the students and the physical character- 
istics of the plant) . If students in school "1" are 
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performinq at 25 percent above the average for schools in 
classification "A," based upon "hard to change circumstances," 
the school efforts may be considered comparable to those in 
school "2" whose students are performing at 25 percent above 
the average for schools in classification "B." 

It follows that a project which has the same relative posi- 
tion in the distribution of schools which have similar 
characteristics as that of another project (which may be 
from a group of scnools with different characteristics) 
could be assumed to have the same degree of effectiveness. 
In other words, effectiveness must be in terms of the per- 
formance which can be reasonsdaly expected, given the known 
circumstances of a project which limit the range of student 
achievement. 

8. Tentative Committee Output 

From a topology of schools, develop a profile for each classi- 
fication showiog " expected values " (mean achievement and 
variance) for school factors; i.e., inputs, processes used 
and cost levels. For example, by choosing combinations of 
school inputs f^at are alike, the student achievement associated 
with each level of cost could be determined ( a norming 
procedure ) . 

Develop a prediction model, incorporating "hard to change" 
elements (including costs) . Schools with higher than pre- 
dicted outputs are said to be more cost effective ( a rating 
scheme ) . 

Develop an explanatory model - In addition to "hard to change" 
elements, the model could include elements that can be changed; 
i.e., types of staff, equipment and techniques. The purpose 
would be to determine the marginal effect of the program ele- 
ments in terms of cost effectiveness ( information for the 
purpose of modifying projects ) . 

5. Output Measures for Modeling Overall Project Effectiveness 

Traditional testing measures where an individual student is in 
terms of a hypothetical national population. The dimensions 
measured are very broad, such as reading achievement (with sub- 
categories of paragraph meaning and work meaning) . It is the 
current practice of compensatory programs to aggregate these 
measures of individuals for use in assessing the overall effec- 
tiveness of individual projects. 

Paragraph Meaning 

Traditonal Tests 

The Stanford Achievement 
Test is an example of an 
output measure for State 
reading programs 

Word Meaning 
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Dr. Mccormick suggested the Measurement Sub-committee might 
assume these conventional tests, when appropriately weighted 
on the basis recommended by the Scaling Sub-committee, could 
form part of the basis for rating the cost effectiveness o f 
individual projects . "~ 

Ou tput Measures for Modeling the Effectivenes s of Specific 
School P"r act ices ^~ 

Study by the Measurement Sub-committee shows two newer purposes 
for testing: (1) measuring stuo ;nt performance (diagnosis) 
for the purpose of assigning treatment, and (2) assessing the 
effectiven ss of the educational treatment prescribed. Both 
require measurement ( criterion-referenced ) in terms of student 
performance according to "instructional objectives." 

Criterion-Referenced Tests 

Student performance based upon 
narrow dimensions of what the 
school is trying to do is used 
in schools for planning and 
evaluating reading instruction. 



Dr. Mccormick feels that attempts to use traditional testing for 
these newer purposes seem doomed to failure because such tests 
were constructed for other purposes. Further, according to 
measurement specialists, the tests are not sensitive to teaching 
strategies . 

Compensatory projects presently use performance objectives 
(student-referenced) for planning instructional strategies 
but not for the evaluation of the overall effectiveness of 
the program. Dr. McCormick recommended that the Committee 
study the feasibility of using criterion-referenced testing 

for measurement of the effectiveness of specific school 

practices. 

A Plan for Developing the Information Necessary f o r Estimatin g 
the Effectiveness of Spe c ific School Practices ~ 

In meeting with the Sub-oommittee on Measurement, 

Dr. Garth Sorenson (UCLA) suggested a plan which Dr. McCormick 

feels may be effective for obtaining the information needed 

for revising projects which have been rated low in effectiveness . 

Such a plan should include the following elements: 

1. A set of criterion-referenced tests (together with a set 
of rules for selecting these tests) . A criterion- 
referenced test measures the intellectual skills the 
teacher is trying to teach. They will be used in (a) 
measuring what the student knows at the beginning of the 



course/ (b) to check the effectiveness of individual 
lessons, (c) to check what the student knows at the 
end of the course. 

2. A set of repeatable instructional procedures de- 
signed to improve the students' performance on 
criterion-referenced tests. These will have to 
provide for differences among pupils in learning 
speed and motivation. Will need some remedial 
routines . 

3. A set for formative evaluation procedures for 
testing and improving the instruction tools. 
Will include (a) a record of each student's 
progress, (b) monitoring procedures to see that 
teacher used the instructional program correctly, 
(c) comparison groups. 

Dr. Sorenson's plan requires specialized staff and a great deal 
of local school cooperation. Such school operations now exist 
in some school districts and Dr. McCormick feels we should 
consider ways of channeling resources to help then evaluate 
specific school practices in terms of their cost effectiveness. 
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I'RESENTATIQN BY DR. RAY SWEIGERT OP HIS MODEL 



Dr. Sweigert said that one way of viewing benefit/cost in relation 
to PPBS is as follows: 



Program Planning 
Budgeting 



Evaluation 
Accounting 



Benefit 
Cost" 



Where Program Planning is considered as the niimerator of a fraction 
and Budgeting as the denominator, Evaluation determines the results 
of prograra planning after implementation. Out of Evaluation comes 
some indication of benefits that were produced by Program Planning. 
Out of Accounting comes as indication of the actual cost of carry- 
ing out this program plan. The benefit/cost " 
evaluation model, but a reporting model. We 
accounting part of it than the benefit part. The 
determining costs, but determining benefits. The 
emphasize benefits, but without both benefits and 
picture does not emerge. For a unit of measure I 

simple one. It is a single student achieving a single learning ob- 
jective to a criterion level of performance. You add up all instances 
of a single student achieving a single objective, weighting according 
to its relative importance and then divide by the commensurate cost. 

Following are the elements in the model: 



formulation is not an 
know more about the 
problem is not 
model I have will 
costs , the whole 
picked a very 



N = 



B = 



Number of students receiving instruction towards achieve- 
ment of a single objective. 

Percent of studeiils mastering given objective prior to 
instruction. 



F = Percent of students mastering upon completion of instruction. 

F-B = Percent of students who mastered objective during 
instruction. 

N (F-B)^ = Number of students who mastered the objective during 
instruction. 

Iq = the relative weight applied to objective o. 



Grade level may be used as a criterion of desired performance. In 
absence of having any really rational means of setting criteria at 
the present time, we might as well use grade level. 

Putting the above elements together gives us the following benefit/ 
cost index for objective o. 

I N (P-B)o 
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n^Sd ?n tfJ'^ •^''"k'^ ^I'l^^ "'^'^^^ P^^'^^ information to be 

used in analyzing benefits versus costs. 

To add across objectives, we introduce a summation sign Z 

o 

C I N (P-B) 



o o o 
t 



C 

To add across groups of students, we introduce a subscript p after N 



^ Iq N (F-B) 
o t P 



and introduce a second summation sign in the following manner 

X z 

P ° Qt P ° - This is the benefit/cost formula 

C 

L^ttl l^tl "^^'""^ determine what is going on in education 

in the State, we are going to have to use something like this in re- 
porcing results. The index provides the relative number of uni2s of 
student benefit per dollar spent. "unioet or unics or 

Dr. Sweigert also cited Hinrichs and his three types of problems 
volving cost/benefit analysis. prooxems 

1. The amount of money is fixed, and you want to see how 
much you can get for that amount of money. Alterna- 
tives with different levels of benefit are examined. 

2. You have a given objective, a given set of benefits 
and you want to achieve it at minimum cost. This is 
generally known as cost effective analysis. Alterna- 
tives with different costs are examined. 

^' ^onL^r^'^i^^.^^ This is the most 

general situation. Alternatives have different bene- 
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The model which was accepted by the Committee for furtht-r develop- 
ment was one presented by Mr. Allen Ginsburgh at the meeting of 
January 21, 1971. It was reproduced here from the minutes of that 
meetiTig, 

STATUS AND SUBTASK REDEFINITION 
An Analysis by Mr. Allen Ginsburgh 



In this section, the status of our progress toward a cost effective- 
ness model will be restated in broad and general mathematical terms. 
The tasks of our sub-committees in Phase II, Component Studies, will 
become more clearly focused. This should permit more detailed 
planning, scheduling and budgeting of these subtasks and, hence, a 
reevaluation of the overall task. 

1. For our purpose, we will consider money spent in Compensatory 
Educational Programs (particularly Title I, Title III and 
Miller-Unruh) as directed toward those portions of the student 
population which statistically are not achieving the same edu- 
cational progress as the overall population (due to poverty, 
ethnic differences or otherwise) . The programs supported in 
Compensatory Education are, therefore, intended to achieve 
additional educational gains in the deprived under-performing 
populations so as to provide them the necessary background so 
that they may also achieve the same gains from each educational 
year as the overall population. The task of our Committee \o 
to measure the cost effectiveness ( a ) of each program (B) 
under these specific Acts in making measurable progress. 
Recognizing the need to make the task more tractable, the 
Committee has limited itself to consider only gains in reading 
and mathematics. Then the cost effectiveness of Program B 
which represents any program sponsored by the Acts (Title I, 
Title III or Miller-Unruh) is 

Measurable Gains in Reading and 
a B= Mathematics Achieved by Program B 

Dollar Cost of Program B 

2. We recognize that there are many measurable attributes which 
may be considered in gaining improved proficiency in mathe- 
matics and reading. These are such things as "word recogni- 
tion," "word comprehension," "sentence comprehension" in 
reading, or "counting by rote" and "simple addition" in mathe- 
matics. We have been given samples of these as lists of detailed 
instructional objectives. 

We must now compile and adopt a set of these "Detailed Instruc- 
tional Objectives" which are individually measurable and which, 
in total, describe what we mean by the "reading skill" and the 
"mathematical skill." For purposes of notation, we will call 
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(a) - 



A representative "measurable 
instructional objective." 



We are going to measure the (a) capability in a population by 
a test before we submit the population to Program B and iden- 
tify the result as (aj) . We are going to retest the population 
again for this same measurable instructional objective after 
we have conducted Program B and identify the result as (ajj) . 
The improvement in this specific "measurable instructional 
objective" due to Program B is 



Above, we identified a number of such "measurable instructional 
objectives" which we can identify as ai, a2, a3 , etc., up to 
an amount to be determined which will describe what we mean by 
the totality of the reading and mathematics skills. For con- 
venience, we will refer to any one of these as the (i)th one 
and consider for a population of (n) students that 



as the total improvement in (i)th detailed instructional 
objective due to Program B. ( (i) is the index of a measurable 
instructional objective.) The identification of all of the 
"measurable instructional objectives" of interest to us is a 
task of our Objectives Committee. 

3. The manner in which we measure ^ Bi is the task of our Methods 
Sub-Committee. We must choose or devise a test or set of teats 
which measure (ajj) gi and (aj) Bi so that we can determine 
each A Bi whicii is important to us in determining the effec- 
tiveness of Program B. 

4. We now recognize that the success of any given educational 
program such as Program B is strongly dependent on the charac- 
teristics of the population taught and the environment in which 
it is taught. Typical of population variables of interest are 
"ethnic distribution," "teacher density," "family economic 
status," and "distribution of mother tongue." Environmental 
factors could include "teacher skill and attitude," "school ad- 
ministration attitudes," "classroom lighting" and a host of 
cpthers. These two sets of variables must first be redefined 
into a set which we can measure quantitatively and those which 
cannot be so treated. We will call 



We know that there are a number of each so we will establish 



(ajj) - (aj) . 



n (aji - aj) = A Bi 



g = a test population measurable characteristics 

e = a test population non-measurable characteristics. 



j = index of test population measurable 
characteristics 



1 « index of test population non-measurable 
characteristics 
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so that we can discuss how we intend to treat each gj and 
each e-j^. 

Our Scaling Sub- Committee has the task of identifying these 
characteristics, dividing them between the gj and the e^^ and 
assuring that they are statistically significant. 

The same program, Program b as example, will result in 
different effectiveness, dependent upon the population variables 
just discussed. If we are to compare the effectiveness of 
different programs with different population and environmental 
circumstances, it will be necessary to establish a functional 
relationship describing the effect of each measurable variable 
(g) . These functional relationships we will describe as F(g) ; 
i.e., some yet to be defined relationships involving (g) . In 
general, we must expect that there will be a different function 
for each (g^), although future studies may .simplify matters. 
Therefore, we write Fj(g)4 to infer this difference. Not only 
that, but we must allow that each (gj) may act differently 
with each (a^^) . So to be general, we must write this function 
of (g) as Pij (gj) . 

Each test population will affect each (aj^) through all (g^) of 
significance. It is our thought that this effect may be Multi- 
plicative rather than additive because of the inter-relationship 
(i.e., ethnic effects will be greater when economic effects 
also exist). As a result, we will tentatively choose to operate 
on each A si with the product of all Fij (gj) . 

This product is written 

'^j (Fij (gj) ) 

This functional product of the effect of the measurable 
statistically significant characteristics of the population 
and environment, effectively chamges the scale by which we 
measure the gains obtained in each meas irable instructional 
objective (a^) through the application of Program B to this 
test population. The gain in each (ai) is then 

"TT j (Fij (gj) ) A Bi 

and the total measurable gain in reading and mathematics 
achieved by Program B 

^ '^j (Fij (gj) ) A Bi 

with the exception that we have not accounted for any non- 
measurable characteristics ie) i which we might find to be of 
significance. 
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B'or the moment, we will assume that we can statistically null 
out the effect of the (e)i by choice of the test populations 
used for the evaluation of Program B. This may be handled 
through the procedural task planned within Phase III, the 
Modeling Phase of the total effort. We then can rewrite the 
numerator for our cost effectiveness relationship as 



51 "TTj (f .j (g.) ) A Bi 



Null 



6, The program costs will be made up of the summation of all of 
the cost elements identified on the program. These cost 
elements, (c) , include cost of teachers, cost of teachers' 
aids, cost of constructional material, etc. If we identify 
(m) as an index of this parameter, then the total cost is 



m 



m 



7, The total expression becomes 

H 'TTj (Fij(gj) ) 



Null e- 



'm 



m 



In summary, our task is to: 
1. 



2. 
3. 



Identify all of the "measurable instructional objectives" 
(a^) making up competence in reading and mathematics. 

Identify or otherwise obtain one or more tests which measure 
these measurable instructional objectives. 

Identify all statistically significant test population and 
test environmental characteristics and divide them between 
those which are measurable (g^) and those which cannot be 
feasibly measured (ej^) , 

Determine the functional relationship of the measurable 
statistically significant "test population and test environ-^ 
mental characteristics" (g.) upon each of the measurable in- 
structional objectives (aij, 

5, Identify all cost elements of the Program. 

These tasks must now each be more fully programmed, scheduled 
and budgeted. 



4. 
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A block diagram of Mr. Ginsburgh's model could look like this: 



PROGRAM 



Subject Content 
Teaching Strategies 
Supplies - Equipment 
Facilities 



Teacher 
Characteristics 



Pretest 



Students 



Posttest 



Gross 
Student 
Gains 



Student 
Characteristics 



Adjusted 
Student 
Gains 
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At the meetiag of December 9, 1971, Mr. William Bronson gave an 
overview of modeling in terms of the steps that are necessary to 
develop a model, and then indicated how the work to date on 
Mr. Ginsburgh's model matched the required steps. 

Presentation by Mr. William Bronson to Committee December 9, 1971 

Steps in Model Building 



1. State hypothesis or hypotheses on which the model is to be 
based. 

2. Identify the elements that seem to have a relationship to the 
problems to be dealt with in the model. 

3. For each of the elements identified in step #2, describe its 
relationship to the other elements or to the total model. 

4. Describe each of the elements in quantifiable terms. 

5. State in mathematical terms (formula) the relationship described 
in step #3. 

6. Insert values for each element in the formula and check the 
results for face validity. 

7. Change input data and predict the new output. If hypotheses 
are sound and model is valid, calculated changes of input 
should produce predictable changes of output. 

8. Field test and validate. 

j\n Evolving Model from the Committee on Cost Effectiveness Hypotheses 

(These hypotheses were implied from Mr. Ginsburgh's model and re- 
flect the beliefs of the Committee in regard to the methodology for 
assessing cost effectiveness) 

1. It is possible to compare programs with similar objectives 
and determine both the relative costs and the relative effec- 
tiveness of the programs . 

2. Programs may be ranked according to relative costs or by rela- 
tive effectiveness or by a cost-effectiveness ratio. 

3. With a given program, it is possible to predict the varying 
impact (effectiveness) of that program on student populations 
of varying backgrounds or characteristics. 

4. It is possible to predict the relative impact a program 

will have when implemented by teachers with varying backgrounds 
or characteristics. 
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5. It is possible to identify successful programs and to 
replicate those programs in other districts. 

6. When replicating a given program, it is possible to predict 
the effectiveness of that program with students of varying 
characteristics when irapleraented by teachers with varying 
characteristics . 

Elements t hat Seem to R el ate to the Hypotheses 



1. 


Objectives . 


2. 


Measurement devices . 


3. 


Progrcim description. 


4. 


Student characteristics. 


5. 


Teacher characteristics. 


6. 


Program costs. 



Rel ationship of Elements 

1. Objectives of different programs must be the same if the 
results of different programs are to be compared. 

2. The same or equivalent measuring devices must be used when 
comparing the results of different programs. 

3. Program content, teaching strategies and other program des- 
criptions are required for the replication of a program but 
not for the measurement of the end results. 

4. There are factors in the backgrounds and experiences of student 
populations that will enhance or inhibit their ability to 
profit from exposure to educational programs . 

5. There are factors in the backgrounds, training and experiences 
of teachers that will enheince or inhibit their ability to 
successfully implement particular educational programs. 

6. There are factors in the backgrounds, training and experiences 
of teachers that will enhance or inhibit their ability to 

be successful with particular groups of students. 

7. In comparing program costs, the same items must be used and 
the basis for calculating costs must be the same in order to 
make valid cost comparisons of different programs. 
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DESCRIPTION AND QUANTIFICATION OF ELEMENTS 



1. OBJECTIVES MUST INCLUDE; 

a. SUBJECT AREA 

b. GRADE LEVEL 

C. LENGTH OF TREATMENT (MONTHS) 
d. EXPECTED OUTCOMES 

(1) GAIN IN MONTHS 

(2) GRADE PLACEMENT LEVEL 

(3) SPECIFIED SCORE ON CRITERION REFERENCED TEST 

(4) OTHER SPECIFIC MEASURE 

2. MEASUREMENT DEVICES MUST INCLUDE: 

a. NAME OF INSTRUMENT 

b. FORM, PUBLICATION DATE, ETC. 

c. KINDS AND FORM OF OUTPUT 

3. PROGRAM DESCRIPTORS: 

a. ALL ITEMS NEEDED FOR REPLICATION 

(1) COURSE CONTENT 

(2) LEVEL OF PHYSICAL FACILITIES, SUPPLIES, EQUIPMENT 

(3) STAFFING REQUIREMENTS; TEACHERS, AIDES, SPECIALISTS, 

ADMINISTRATORS 

(4) LEVEL OF SUPPORTING SERVICES 

(5) STAFF DEVELOPMENT REQUIREMENTS 

b. NO ITEMS NEEDED FOR COST EFFECTIVENESS FORMULA 
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4. STUDENT CHARACTERISTICS 

a. LISTING OF RELEVANT CHARAC ?ERISTICS 

b. WEIGHT INDICATING RELATIVE INFLUENCE ON RESULTS 

5. TEACHER CHARACTERISTICS 

a. LISTING OF RELEVANT CHARACTERISTICS 

b. WEIGHT INDICATING RELATIVE INFLUENCE ON RESULTS 

6. PROGRMl COSTS 

a. FORM IN WHICH COST GOES INTO FORMULA 

(1) TOTAL COST 

(2) COST PER STUDENT 

(3) OTHER (SPECIFIED) 

b. ITEMS TO BE CONSIDERED 

(1) (SEE FINAL REPORT OF SUB-COMMITTEE ON COSTS) 

c. METHOD OF DETERMINING EACH ITEM 

d. METHOD OF CALCULATING FINAL FIGURE FOR FORMULA 
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POTENTIAL BENEFITS 



The potential benefits to education that could result from the com- 
pletion of the methodology started by this Committee would include 
the capability to make decisions based on: 

1. The relative cost effectiveness of programs with the 
same objectives when a cost effectiveness index has 
been developed. 

2. Objective data related to costs of programs and 
program elements. 

3. Objective data related to the effectiveness of dif- 
ferent programs with the same objectives. 

4. The impact of a program on different student 
populations . 

5. Staffing requirements in terms of teacher charac- 
teristics, skills or teaching styles required. 

6. The evaluation of different instructional strategies. 

7. The results of replicating successful programs or 
program elements in other locations. 

We may speculate that the impact of these improved decision making 
benefits upon the institution of education in the United States 
could be dramatic. 

School administrators and teachers will have available more discrete 
tools as suggested in items 2 through 6 with which to both develop 
and monitor their various programs. Their potential to do a better 
job and to more effectively adjust to the desires and needs of the 
public will increase. 

The confidence of taxpayers, parents and their elected representatives 
in education may grow through a clearer understanding and better 
documentation of what happens. Clearer descriptions of what our youth 
are learning, how they are learning it, and how efficiently the edu- 
cational dollar is used are all prerequisites for increased public 
support of education. Elements of this methodology such as items 3 
through 7, when incorporated into our institutional efforts to train 
teachers, may increase dramatically the instructional effectiveness 
of teachers. 

The kinds of information generated through the development of con- 
cepts in items 3, 4, 5 and 6 may produce a far more detailed and 
valid approach to objective teacher evaluation than has been possible 
before. 
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These potential benefits: (1) greater public support of educa- 
tion; (2) improved teacher training; and (3) more effective 
teacher evaluation, are among the most significant that could 
accrue to the whole enterprise of education. 



4 
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